Auditing a generative AI system requires more than testing whether the model can answer questions accurately. A production GenAI application may combine a foundation model, system prompts, retrieval-augmented generation, vector databases, user data, external tools, APIs, guardrails, memory, plugins, and human review. Each component can change the risk and the evidence an auditor needs.
The current ISACA AAIA outline gives 46% of the exam to AI Operations and 21% to AI Auditing Tools and Techniques. It explicitly includes AI lifecycle, testing, threats and vulnerabilities, incident response, evidence collection, data quality, analytics, and audit outputs. A useful audit therefore tests the complete system, not only the base model.
Define the GenAI system boundary before testing outputs
Document the model provider and version, hosting arrangement, prompts, retrieval sources, embedding model, vector store, tools, plugins, APIs, identity controls, user groups, logging, monitoring, and downstream actions. Determine which components are managed internally and which can change at the provider’s discretion.
Map sensitive data flows. User prompts may contain confidential data. Retrieval can pull private documents. Tool calls can expose application credentials. Conversation history may be stored for analytics. Provider telemetry may cross organizational or geographic boundaries. The audit boundary must include those flows.
Also classify what the system is allowed to do. A chatbot that drafts text has a different risk profile from an agent that can send email, approve refunds, change cloud resources, or query production databases. Capability and autonomy drive testing depth.
Build an evaluation set tied to business risk
Random prompts are not a sufficient audit sample. Create an evaluation set that reflects real use cases, high-impact decisions, known edge cases, prohibited content, sensitive data, adversarial instructions, and plausible misuse. Include both normal users and attackers in the test design.
Define expected behavior before running the test. Some items have objectively correct answers; others need criteria such as groundedness, policy compliance, refusal behavior, completeness, or escalation to a human. If auditors decide what “good” means only after seeing the output, bias enters the evaluation.
Preserve the prompt, system configuration, model/version, retrieval context, tool results, output, timestamp, and scoring rationale. GenAI behavior can be nondeterministic and providers may update models, so reproducibility requires more context than a screenshot.
Test confabulation and misinformation in context
Generative models can produce fluent statements that are unsupported or false. NIST’s Generative AI Profile uses the term confabulation for confidently stated erroneous or false content, while OWASP’s 2025 GenAI risk taxonomy includes misinformation. The audit should focus on how those failures affect the application’s purpose.
For a low-stakes brainstorming assistant, an incorrect suggestion may be easy for a user to detect. For legal, medical, financial, security, or operational workflows, an invented citation, fabricated control status, or incorrect command can create material harm.
Test whether the application grounds answers in approved sources, distinguishes retrieved evidence from model-generated explanation, exposes uncertainty appropriately, and routes high-risk cases to human review. Measure error patterns rather than reporting one dramatic example.
Probe prompt injection and instruction hierarchy
Prompt injection occurs when untrusted input attempts to alter the model’s intended behavior. Direct injection comes from the user; indirect injection can arrive through retrieved documents, web pages, emails, tickets, or other content the model is asked to process. A GenAI system that reads external data must treat that data as potentially adversarial.
Test whether malicious instructions can override system constraints, reveal protected data, trigger unauthorized tools, change transaction parameters, or bypass required approval. Include payloads in documents and retrieved content, not just in the chat box.
Do not accept “the system prompt says not to do that” as a sufficient control. Strong controls should be enforced outside the model where possible: authorization on every tool call, allowlisted actions, parameter validation, least-privilege credentials, output filtering, transaction limits, and independent approval for high-risk actions.
Audit sensitive-information exposure across the pipeline
GenAI data leakage can occur through prompts, retrieval, training or fine-tuning data, model outputs, logs, embeddings, caches, tools, and administrative interfaces. Test whether one user’s context can appear in another user’s response and whether the model can retrieve documents outside the caller’s authorization.
Data Loss Prevention can detect or restrict sensitive information moving through approved channels, but it is only one layer of GenAI leakage control. It should complement application authorization and data minimization rather than substitute for them.
Review whether prompts and outputs are retained, whether the provider uses customer data for training, how deletion works, and which administrators can inspect conversation history. Search logs for secrets such as API keys, access tokens, customer identifiers, and regulated data.
Test bias and harmful performance differences
Bias auditing should be tied to the population and decision supported by the system. Define relevant groups, expected service quality, and unacceptable disparities before testing. For a résumé assistant, evaluate whether equivalent qualifications produce materially different recommendations. For a customer-support system, compare tone, escalation, and resolution quality across representative users.
Do not assume one global “bias score” proves fairness. Data representation, prompt design, language, geography, accessibility, and deployment context can create different harms. Where protected or sensitive attributes are involved, coordinate with legal, privacy, and subject-matter experts.
Evidence should include the test population, sampling method, metrics, confidence or uncertainty, thresholds, exceptions, and remediation decisions. A claim that “the model was tested for bias” without reproducible evidence is not an auditable control.
Audit agentic actions and retrieval augmentation
GenAI becomes higher risk when it can act. Review each tool the model can invoke, the permissions behind it, allowed parameters, transaction limits, and whether human approval is required. A model that can read a calendar is different from one that can delete records or transfer money.
Test whether the model can chain individually permitted actions into an unintended outcome. An agent might be allowed to read customer records and send email; together, those abilities could enable data exfiltration. Least privilege must therefore consider action combinations, not only individual API permissions.
Audit logs should show the model or agent identity, the user or process that initiated the task, tools called, parameters, authorization decisions, results, and approvals. If the organization cannot reconstruct an automated action, it cannot investigate or govern it.
Review retrieval, embeddings, and source integrity
Retrieval-augmented generation can improve grounding, but it introduces a data pipeline that needs security and quality controls. Test document ingestion, source approval, access filtering, chunking, embedding, indexing, deletion, and reindexing. A deleted source should not remain indefinitely retrievable because an old embedding was never removed.
Authorization must be enforced when retrieval occurs, not only when documents are initially indexed. If users with different permissions share one vector store, the retrieval layer must prevent a user from receiving chunks they could not access in the source system.
Poisoned or malicious documents can also manipulate outputs. Treat retrieved content as untrusted input and test whether embedded instructions can alter behavior or trigger tools.
Audit monitoring, change management, and incident response
Model behavior can change because prompts, retrieval data, provider versions, safety settings, tools, or business use cases change. The organization should have a baseline and trigger re-evaluation when material changes occur. Silent provider updates deserve particular attention in high-impact applications.
Operational metrics may include refusal rates, groundedness, unsafe-output events, prompt-injection detections, tool-call failures, data-leakage alerts, latency, cost, human overrides, and drift in evaluation scores. Thresholds should trigger named actions rather than simply populate a dashboard.
Incident response should cover containment of the AI capability itself. Teams may need to disable a tool, revoke credentials, switch models, remove a poisoned retrieval source, block a prompt pattern, preserve logs, notify affected parties, or suspend the service. Practice those actions before an incident.
Cost and availability also belong in the control environment. Unbounded prompts, recursive tool use, large retrieval requests, or runaway agents can create unexpected consumption and denial-of-service conditions. Test quotas, timeout behavior, concurrency limits, budget alerts, and the ability to stop an abusive or malfunctioning session without taking the entire service offline.
Where the application uses content filters or policy classifiers, audit their placement and failure behavior. A filter that runs only on the final answer may miss unsafe tool parameters or sensitive retrieval before generation. Conversely, aggressive filters can block legitimate business use and encourage users to bypass the approved system. Measure both control effectiveness and operational side effects.
Human review should also be tested for automation bias. Give reviewers cases where the model is deliberately wrong but persuasive and measure whether they challenge it. Review interfaces should expose enough source context and uncertainty for a person to make an independent decision; a nominal approval button does not create effective oversight if the reviewer has no practical basis to disagree.
Make audit evidence reproducible and decision-ready
GenAI auditing is difficult when evidence cannot be replayed. Preserve configuration snapshots, provider documentation, evaluation sets, prompts, retrieved context, tool traces, scoring methods, reviewer decisions, exceptions, and change history. Where outputs are nondeterministic, run repeated trials and report distributions rather than one cherry-picked response.
Use NIST and OWASP taxonomies to make sure the test plan considers current risk classes, but do not turn the audit into a checklist detached from business impact. The same prompt-injection weakness can be minor in a sandbox and critical in an agent with production privileges.
Across ISACA certifications, the defensible conclusion comes from the system’s objectives and evidence: whether GenAI risks were identified, controls were designed appropriately, testing demonstrates operating effectiveness, failures are monitored, and residual risk is owned. Auditing the model alone is not enough; audit the application, data, tools, people, and lifecycle around it.