ISACA AAIA: Auditing Artificial Intelligence with Reliable Evidence

The ISACA Advanced in AI Audit (AAIA) certification is designed for experienced audit and advisory professionals who evaluate AI systems, governance and operating controls. ISACA's published 90-question exam has three domains: AI Governance and Risk (33%), AI Operations (46%) and AI Auditing Tools and Techniques (21%). It also establishes eligibility through CISA or specified qualifying professional designations with the relevant audit or advisory role focus. Unlike a general introductory AI course, the credential asks whether an auditor can evaluate a system's risks and controls and produce defensible assurance. The focus is on evidence, professional skepticism and accountability throughout the AI lifecycle.

Establish the AI system's purpose and audit boundaries

An audit begins by identifying the system under examination: who owns it, what decisions it influences, which data it uses, what components are internal or external and what business outcomes it is expected to support. AI deployments may involve a pretrained vendor model, internal retrieval index, orchestration service, application code and human review. Auditing only the model while ignoring the surrounding workflow can miss serious risks. A retrieved document may leak data through weak permissions even if the model's base weights are well protected.

Define scope using risk and materiality. A document drafting assistant for internal low-impact text presents a different assurance challenge from a model that recommends loan decisions or patient triage. The auditor should map affected people, financial and reputational impact, compliance exposure and operational dependencies. Document what is outside scope and explain why. A vague scope such as “review AI security” offers little basis for adequate testing or meaningful conclusions.

Audit objectives should be framed as testable statements. For example: all production models have identified owners; sensitive training data follows approved permissions; consequential decisions have defined human oversight; changes to model versions are reviewed; and incident handling covers model-specific failures. Collect evidence that demonstrates whether these conditions operated during the review period rather than relying solely on policy claims.

Evaluate governance, standards and AI risk management

Governance responsibilities include approval, policy, model inventory, vendor evaluation, data protection and control monitoring. An audit should determine whether committees and owners have actual authority to act. Meeting minutes alone do not prove governance works if high-risk findings are repeatedly deferred without accountable decisions. Compare policy requirements with sampled production systems and approvals. Ask whether exceptions are documented, time-limited and reviewed as conditions change.

AI risk assessment should consider fairness, privacy, reliability, safety, security, explainability and misuse where relevant to the use case. Different systems require different evaluation methods. A fraud detection model may have important false-positive consequences; a generative assistant may hallucinate unsupported content or follow malicious instructions embedded in retrieved data. Risk scoring should identify assumptions and affected groups. An AI governance audit connects documented risk tolerances to evidence that the organization actually approves, monitors and remediates high-impact AI uses.

Standards and legal obligations provide references but are not replacements for audit judgment. Determine which framework requirements actually apply and whether controls are implemented to meet them. Compliance with a checklist can coexist with weak operational performance. The auditor should disclose when evidence is incomplete or when a benchmark is not representative of real users.

Audit data management and model development

Data governance underpins AI performance and legitimacy. Examine data sourcing, permission, provenance, quality, labeling, retention and access. The auditor should determine whether datasets are appropriate for the stated purpose and whether transformations introduce bias or errors. A training dataset can be statistically large but systematically exclude important populations. Sample critical data flows and verify that approvals match actual use, especially for external datasets and data obtained from third-party processors.

Model development introduces requirements for reproducibility and controlled change. Review versioned datasets, training configuration, evaluation records, release approvals and dependency management. If a hosted vendor model changes without notice, determine whether the organization's tests detect material differences. Assess separation of development and production environments and limit who can publish new versions. A claimed approval control is weak if engineers can bypass it through an undocumented deployment interface.

For generative AI, assess how prompts, retrieval context, tool permissions and output handling interact. A model that answers accurately in a static test may fail when supplied a malicious document or an unexpected user request. Test whether source authorization is enforced outside the model and whether consequential actions require independent checks. When auditing generative AI systems, evaluate model behavior together with retrieved-document permissions, tool execution boundaries and the application's independently enforced authorization.

Test operations, monitoring and incident handling

AI operations should have meaningful performance objectives, ownership for model monitoring and predefined escalation paths. Infrastructure uptime alone cannot demonstrate that an AI system continues to make appropriate decisions. Model drift, data pipeline errors, label delays, unusual prompt patterns and harmful outputs may all require investigation. Determine which indicators are monitored, who reviews them and what threshold triggers action. A dashboard without documented ownership or response evidence provides limited assurance.

Testing should examine representative inputs and edge cases. Where possible, evaluate error distribution across relevant groups, changes over time and the severity of incorrect decisions. Some outcomes may not be known immediately; a credit decision might take months to validate against actual repayment behavior. Audit whether the organization uses appropriate proxy metrics without presenting them as verified final outcomes. When automated evaluation judges are used, assess their calibration and limitations.

Incident response for AI includes data leaks, security compromise, model malfunction and harmful decisions. Determine whether staff can disable or restrict a model, revert safely to a previous version and maintain critical services. A playbook should identify evidence preservation, owners and notification assessment. An incident that changes model outputs without stopping the system may be particularly difficult to detect; auditing should test the organization's ability to identify such silent failures.

Use audit tools, sampling and evidence critically

Sampling must support the audit objective. A small collection of successful demonstration prompts does not prove that a model handles real-world edge cases safely. Select samples based on risk, relevant populations, major deployment changes and known incidents. Record the dataset, evaluation criteria, environment and model version for reproducibility. If access restrictions limit testing, describe the resulting assurance limitation rather than filling the gap with unverified confidence.

Auditors may use AI tools to speed document comparison, anomaly analysis or control testing, but the tools introduce their own risks. A generated audit summary may omit relevant exceptions or state that a missing document was reviewed. Require human validation of important findings and maintain the underlying source evidence. Protect confidential audit material from unauthorized tool ingestion and avoid disclosing sensitive data in prompts to external systems without approval. The practical use of AI in internal audit must preserve objectivity and traceability.

Audit findings should state criteria, condition, evidence, cause where known, impact and a practical recommendation. Classify severity according to business effect and residual risk rather than describing every missing document as equally critical. A constructive report enables management to address the underlying control weakness and gives the audit function a credible basis for follow-up testing.

Build an audit finding from incomplete AI monitoring evidence

An organization operates a loan-assistance recommendation model and claims to monitor its fairness monthly. The audit file contains dashboard screenshots showing aggregate accuracy but no evidence of population-level review or of action when the distribution changes. An AAIA auditor should identify which governance requirement applies, what assurance the current evidence provides and which limitations remain. A screenshot proves that someone displayed a metric; it does not establish that the metric was appropriately defined, reviewed or acted upon. Request documented thresholds, ownership, review records and results from representative evaluation data.

Suppose the team explains that final loan outcomes take months to become known. That delay is a legitimate constraint, but it does not eliminate the need for monitoring. Evaluate whether proxy measures are suitably validated and whether periodic outcomes testing occurs once labels arrive. If sensitive attributes cannot be used in operational prediction, determine whether permitted audit processes can still assess discriminatory impacts under applicable rules. The auditor must distinguish an actual control failure from an evidence gap while communicating the associated risk.

A defensible finding describes criteria, condition, cause where supportable, consequence and a feasible recommendation. It might require risk-based evaluation by customer segment, documented response to drift and periodic independent validation. Management should be able to demonstrate remediation through completed review records and measured outcomes rather than a revised policy document alone. This exercise links AI governance, operations and audit technique—the three AAIA domains—through evidence-oriented professional judgment.

Practice a complete AI assurance engagement

Choose a customer-service assistant that retrieves internal documents and can initiate limited account changes. Define its intended use, data inventory, vendor dependencies, control owners and high-risk failure scenarios. Review access control, logging, model evaluation, prompt-injection defenses and change management. Design samples that test both ordinary service quality and adversarial behavior. Then produce a finding where a retrieved document is accessible outside the intended tenant, showing the evidence, affected control, risk and remediation test.

Repeat the exercise for a model that predicts maintenance failure and has delayed ground-truth results. Decide how the team should monitor proxy signals while waiting for validated labels. Challenge whether the existing alert thresholds are meaningful and whether changes to the training dataset receive approval. These cases connect all three AAIA domains rather than treating governance, operations and testing as separate memorization lists.

ISACA's AAIA is distinct from the IAPP AIGP credential, which broadly addresses responsible AI governance. AAIA emphasizes audit evidence and assurance by qualified professionals. Use ISACA's current eligibility page, exam content outline and candidate guide as the authoritative sources for registration, domain weighting and current program rules.