Advanced in AI Audit (AAIA) — Lesson 2: AI Risk, Data, Models, Development, Validation, and Documentation

Learning Objectives

• Explain the roles, information, criteria, dependencies, and decision points governing AI risk assessment and training data.
• Implement or assess model development, model validation, and model documentation using controlled evidence and acceptance criteria.
• Evaluate the condition where a control passes a basic check but model validation fails because model development data, criteria, or dependency assumptions differ and select a proportionate corrective response.

Detailed Topics

• AI risk assessment applied to the course scenario, expected behavior, stakeholders, controls, risks, and evidence.
• training data applied to the course scenario, expected behavior, stakeholders, controls, risks, and evidence.
• model development applied to the course scenario, expected behavior, stakeholders, controls, risks, and evidence.
• model validation applied to the course scenario, expected behavior, stakeholders, controls, risks, and evidence.
• model documentation applied to the course scenario, expected behavior, stakeholders, controls, risks, and evidence.

Instructor Demonstration

The instructor traces an audit function evaluating an AI-enabled business process for governance, data, models, security, ethics, operations, and accountability. The walkthrough connects frameworks, processes, tools, decisions, and evidence captured in the AAIA audit universe, risk assessment, AI control tests, and assurance report. It also tests the exception where a control passes a basic check but model validation fails because model development data, criteria, or dependency assumptions differ.

Hands-On Exercise

Complete a controlled ISACA-aligned exercise covering AI Risk, Data, Models, Development, Validation, and Documentation and preserve the verified result in the AAIA audit universe, risk assessment, AI control tests, and assurance report.

1. Confirm scope, roles, criteria, prerequisites, information sources, dependencies, and acceptance conditions.
2. Execute the governance, risk, audit, security, data, cloud, or technology workflow while recording decisions and evidence.
3. Test normal behavior, one boundary condition, and the stated exception using a controlled scenario.
4. Document findings, corrections, ownership, retest criteria, communication, and operational handover.

Scenario Analysis

• Identify which role, process, objective, control, data source, or decision most strongly influences AI risk assessment.
• Compare two defensible responses and explain governance, risk, audit, security, privacy, operational, and business trade-offs.
• Determine how the AAIA audit universe, risk assessment, AI control tests, and assurance report should be approved, retained, reviewed, retested, and updated.

Troubleshooting Guidance

Start where expected behavior first differs from the observed result. Check scope, criteria, ownership, data quality, process sequence, policy alignment, version compatibility, dependencies, status, and evidence separately. Treat “a control passes a basic check but model validation fails because model development data, criteria, or dependency assumptions differ” as a hypothesis until records, interviews, configurations, calculations, and retest outcomes confirm the cause.

Validation Checklist

• Required roles, processes, criteria, controls, information, approvals, and dependencies are present.
• The AAIA audit universe, risk assessment, AI control tests, and assurance report connects each conclusion to objective, reviewable evidence.
• Normal processing, a boundary condition, dependency behavior, and the stated exception were tested.
• Corrections, ownership, retest criteria, documentation, communication, and operational handover are complete.

Lesson Outcome

Learners can apply AI Risk, Data, Models, Development, Validation, and Documentation, explain the resulting governance, risk, audit, security, data, or operational behavior, and defend the outcome through the AAIA audit universe, risk assessment, AI control tests, and assurance report.