ISTQB CT-AI: Testing Machine Learning and AI-Based Systems

The ISTQB Certified Tester AI Testing (CT-AI) qualification focuses on testing systems that incorporate artificial intelligence. It asks different questions from a course about using an AI assistant to write test cases. The tester must examine data quality, model behavior, performance measures, uncertainty, bias, robustness and changes during the AI lifecycle. ISTQB now identifies CT-AI v2.0 as the latest syllabus; v1.0 remains in a staged transition, with English v1.0 examinations due to retire in April 2027 and non-English versions later that year. The Foundation Level (CTFL) is a prerequisite for the newer certification. Candidates should follow the current ISTQB version-specific syllabus and exam-provider arrangements rather than assume a course labeled merely “AI Testing” uses the right release.

Understand what makes AI systems difficult to test

Conventional software often allows a direct expected-output assertion for a defined input. A machine-learning component may return probabilities, rankings or generated responses, and its behavior can change when data distributions or model versions change. Testing still requires clear requirements, but success criteria may involve statistical performance, fairness, reliability and unacceptable failure types rather than identical text outputs. A classifier with 96 percent overall accuracy might be unusable if it fails on the rare high-risk cases that matter to the business. The tester must understand the model's purpose and data context to interpret that result.

Create test oracles that correspond to real outcomes. For a spam filter, false positives and false negatives have different consequences; for a medical triage system, missed urgent cases may be especially costly. Distinguish functional performance from qualities such as explainability, robustness, security and data protection. Because AI models can be probabilistic, repeated evaluation may be needed to estimate consistency. Test reports should describe confidence and sample limitations rather than present one lucky model run as proof of reliable operation.

Assess training, validation and live data quality

AI quality begins with data. Inspect provenance, labeling rules, duplicates, missing values, class balance, leakage between training and evaluation sets, and representation of relevant populations. If the same customer appears in both training and test samples in a way that leaks target information, apparent model performance may exaggerate real capability. A model trained on last year's transactions may struggle when business patterns change or a new customer segment appears. Test data needs to reflect plausible use, not simply repeat convenient training examples.

Data transformation code is part of the system under test. Units, categorical encoding, handling of missing values and text normalization can alter features silently. Test those pipelines with edge cases and validate assumptions before interpreting model metrics. For privacy-sensitive workflows, use controlled and representative datasets that do not expose protected records unnecessarily. The work of AI data privacy supports this discussion where production-like evaluation would otherwise create an uncontrolled copy of personal information.

Use meaningful metrics and test design techniques

Select measures that suit the task: precision, recall and F1 score for relevant classification problems; suitable error measures for regression; ranking or retrieval measures where outputs are ordered. Examine threshold trade-offs and confusion matrices instead of treating a summary percentage as sufficient. A fraud detector tuned for high recall may generate more false alarms; the business must decide whether that trade-off is acceptable given review capacity. Explain the meaning of each metric in the system's operational context.

Testing techniques may include equivalence classes, boundary analysis, metamorphic testing, pairwise combinations, A/B comparisons and back-to-back testing. Metamorphic testing is valuable when the exact output is unknown but a relationship should hold: for example, changing an irrelevant field in an image should not completely reverse a stable classification. Tests should also exercise unseen environments, corrupted input, rare language variants or shifting distributions. Document the expected relationship and meaningful tolerance before examining results to avoid redefining success after failure.

Test robustness, adversarial behavior and security

AI services can fail through adversarial inputs, poisoning, prompt injection, excessive privilege or unexpected downstream actions. Separate attacks on model training from attacks during inference and from ordinary application vulnerabilities. A vision model could misclassify after an imperceptible image perturbation; a generative assistant might obey instructions embedded in a retrieved document; a data pipeline might import malicious records. Each case needs a test strategy aligned to its attack path and potential business impact. Security testing should not rely only on whether the model refuses one example prompt.

When an AI component has tools, tests must verify independent authorization and output validation. A helpful generated command should not execute with a privileged service identity merely because it appears plausible. If a model can recommend purchases, distinguish recommendation from the final approval control. Measure whether the surrounding application correctly enforces limits when the model produces incorrect or maliciously influenced outputs. Model and conventional software controls are complementary, and defects may originate in either layer.

Organize testing around model and software lifecycles

Version control must cover datasets, training configurations, model artifacts, prompts, dependencies and deployment settings. A model update can change performance even when the API is unchanged. Establish baseline evaluations before promotion, compare new and old versions on representative cases, and define conditions for rollback. Test data drift and monitoring after release because live input distributions can move outside the training environment. Operational evaluation should track degradation, unusual error patterns, response latency where relevant and user impact.

An organization's AI testing plan must also consider human oversight. For an assistant summarizing financial statements, a review control may catch major numerical errors, but only if reviewers can see source passages and have time to challenge suspicious claims. Rehearse conditions where the model should abstain or escalate. The tester should ask whether these safeguards work under load, with incomplete source records and during vendor outages. This perspective connects quality assurance to actual operating capability rather than an isolated benchmark report.

Walk through a customer-support classifier case

A service provider wants a classifier to route customer complaints into standard, urgent and legal categories. Begin by obtaining the intended definition of each category and identifying the highest-consequence errors. Inspect whether historic labels were consistent and whether uncommon complaint types are represented. Split training and evaluation records so repeated conversations from the same customer do not leak across sets. Establish a confusion matrix, class-specific metrics and acceptance thresholds agreed by the business before model selection.

Next introduce realistic edge cases: mixed-language complaints, sarcasm, new product names, partially redacted records and simultaneous legal and technical issues. Test how the system behaves when confidence is low or required information is missing. A safe implementation may route uncertain complaints to trained staff instead of silently forcing a category. After deployment, monitor drift in complaint topics and the rate of manual corrections. This case makes clear why AI test design is about measured risk, data representativeness and operational fallback, not solely synthetic benchmark scores.

Validate an AI triage model after the data distribution changes

A hospital deploys a classifier to route incoming service requests to specialist teams. During validation it performs well on historical tickets, but the live environment introduces new terminology, language variants and a different ratio of urgent cases. High aggregate accuracy can hide poor detection of the rare urgent category. Define an evaluation set that reflects expected deployment conditions and preserve a holdout set independent of model training. For a high-impact class, examine recall, precision, confusion matrices and the operational costs of false negatives and false positives. Discuss whether threshold adjustment makes sense and who can authorize a trade-off that affects patient response.

Inspect data provenance and leakage. If the training examples contain the final routing label inside a comment added after triage, a model may appear much stronger in offline testing than it can be in production. Duplicated requests from one incident can also spill across training and validation splits, making generalization look deceptively good. Test for missing fields, unexpected character sets and values outside the learned range. Stratify results by relevant language or workflow segments where doing so is lawful and meaningful, and record sample sizes so a seemingly dramatic percentage does not hide a handful of observations.

Performance monitoring should detect meaningful distribution changes without confusing every shift with a model failure. Track both input drift and delayed outcome labels. A monthly change in ticket categories could reflect seasonal demand, a new application or a genuine deterioration in prediction. Use shadow evaluation or controlled rollout to compare a candidate model with the deployed one, ensuring that changes in measurement are documented. When human reviewers override routing, capture their reasons and investigate patterns rather than assuming every override is correct or every model disagreement is an error.

Security and robustness must accompany ordinary testing. A malicious user might craft text that causes low-priority routing or leaks information in generated summaries. Build adversarial and malformed-input cases, verify that access controls protect the underlying records and define fallback behavior if the model becomes unavailable. A CT-AI strategy connects data quality, system behavior, monitoring and human consequences across the lifecycle. Passing one benchmark on a clean static dataset does not demonstrate safe operation after the environment changes.

Distinguish CT-AI from testing with generative AI

The ISTQB Foundation Level establishes common testing concepts needed before CT-AI. The adjacent CT-GenAI certification focuses on using generative AI in testing workflows—such as generating and evaluating test ideas—whereas CT-AI concentrates on the quality of AI-based systems themselves. There is overlap when a tester evaluates an LLM application, but the primary question differs. Plan your preparation around the current CT-AI v2.0 learning objectives, sample materials and version notices, and make sure every technique can be explained using a realistic model, dataset and risk scenario.