A reliable Claude agent must prove more than the quality of its final answer. Its tests should verify which tools actually ran, what permissions were enforced, how uncertain side effects were handled, and whether the evidence supports each decision. This walkthrough develops a practical evaluation and release-gate methodology for those workflows, including controlled failure injection and checks that cannot be replaced by an AI-generated reassurance.
Why an agent needs a release test, not just a convincing demo
An agent can produce an excellent answer in a demonstration and still fail as a production system. A natural-language reply may sound correct while the agent has consulted the wrong account, repeated an irreversible tool call, bypassed an approval step, or omitted evidence that a reviewer would need to defend the outcome. Testing therefore has to cover both what the assistant says and what the surrounding application actually permits and records.
Anthropic’s evaluation guidance begins with concrete success criteria, representative test cases, and grading methods suited to the behavior being measured. For a simple classifier, exact-match labels may be enough. For a tool-using workflow, the successful outcome often includes a trace: which data was available, which tool was requested, what the executor returned, whether a policy gate intervened, and which externally observable state changed. A grader that looks only at the final reply can miss a serious violation in the middle of the run.
Consider a fictional customer-support agent. It may explain a refund policy, consult an order service, request a supervisor’s approval, and prepare a refund. The difference between "eligible for review" and "refunded" is a business-state transition, not a difference in tone. A release test must catch an agent that reports a completed refund when the payment endpoint actually timed out.
Begin with a task and a failure budget
Build a small inventory of actions and their consequences before choosing an evaluator. Reading public documentation is a low-consequence operation; reading a protected customer record requires identity scoping; issuing a payment has a direct side effect. These activities need different success criteria. The policy for one fictional workflow might be: no customer data may cross tenant boundaries; a refund above a defined threshold always requires external approval; a timeout never counts as a receipt; and the final answer must distinguish completed, rejected, pending, and unknown outcomes.
Write the expected outcomes as application assertions. "The model should be helpful" is not testable. "No payment tool executes without a valid approval record tied to this order, amount and authenticated actor" is testable with a mocked executor. "When an external transaction is uncertain, the customer-facing response reports pending verification rather than success" can be checked against both a trace and a response rubric. Define the severity of a violation and the action required when it occurs. One critical authorization failure can block release regardless of a high average score on routine answers.
A test plan also needs a budget for false alarms. Overly conservative automation can deny legitimate actions or route every harmless case to human review. For each failure category define who is harmed, how it is detected, and what trade-off the team is willing to make. An escalation policy that always says "ask a person" may look safe while making the product unusable.
Build a small but discriminating scenario collection
A useful initial suite for the fictitious support agent could contain a clean order lookup, a refund request below approval threshold, a high-value request needing supervisor approval, an order belonging to another customer, a missing order, an expired policy, a contradictory pair of policy documents, a payment request that times out, a retry after an uncertain result, and an attempt to hide a malicious instruction inside a retrieved note. Each test should have input fixtures, authorized actor, expected available tools, expected allowed state changes and what kind of response is acceptable.
Do not collapse a scenario into a single ideal sentence. A test often has several acceptable phrasings but only one acceptable transactional result. Record expected outcomes separately from exact text. In a refund-timeout case, the most important assertion may be that the payment service was not called a second time without reconciliation; the agent’s wording can then be judged for truthful uncertainty. Keeping text and action criteria separate makes a future rewrite of the reply style less likely to mask a changed security behavior.
Vary the cases by cause, not cosmetic wording. Ten almost identical orders with different amounts provide less coverage than a set containing permissions, missing evidence, stale policies, rate limits, duplicate requests and partial tool execution. Use synthetic account numbers and sample documents. Never place actual customer credentials, secrets, production refunds or private transcripts into a public tutorial or an uncontrolled test runner.
Grade the workflow with three complementary methods
Deterministic checks should own assertions for object identity, schema validity, forbidden operations, idempotency keys, approval receipts and known system state. When a test denies a cross-tenant lookup, inspect the mocked tool call and resulting storage state. Do not award success because the model politely apologized after attempting an unauthorized request. The trusted executor should refuse the request, and the test should observe that refusal.
Human reviewers are still valuable for nuanced judgments: whether the response is understandable, whether a cited passage actually supports a conclusion, and whether an escalation message gives the right information. Define a rubric with observable descriptors. "Excellent source reasoning" is too vague; "names the source version, identifies the unresolved contradiction, and withholds a final policy decision pending review" gives reviewers concrete criteria. Record agreement between reviewers on a small calibration sample before relying on their scores.
LLM-assisted graders can help sort larger sets, but they are another model-mediated component, not an independent truth oracle. A grader may prefer fluent prose, ignore missing citations, or be influenced by instructions embedded in the answer it is assessing. Compare grader decisions to known human-reviewed examples; provide original source evidence where it is safe to do so; and require deterministic checks to overrule an apparently favorable model score when an authorization rule fails. Separate graders from the agent’s own self-justification so the second pass does not merely endorse the first.
Failure injection reveals what ordinary samples cannot
A realistic agent test runner should be able to return deliberately malformed tool results, delayed responses, expired tokens, empty evidence, contradictory documents, repeated callbacks and incomplete downstream transactions. These are controlled fixtures, not real attacks on an external provider. The test harness can cause a payment mock to persist a transaction and then simulate a lost acknowledgement. An unsafe agent immediately retries; a safe application checks the transaction identifier or requests human reconciliation before sending another write.
Fault injection should specify whether the failure occurred before execution, during execution, or after a side effect was committed. The distinction is important for a client-executed tool loop: the model can propose a tool call, but only the host and external service know whether an operation actually happened. An integration test should preserve the unique tool-use correlation and the external business operation ID. Do not treat a repeated natural-language request as a fresh transaction automatically.
Also test interruptions between workflow steps. If the agent has collected facts and then an execution process crashes, a resumed run must not infer that all subsequent approvals or writes already occurred. Persist the minimum durable state needed to establish what is known. A checkpoint or summary without transaction evidence is not proof of completion. This test area complements, but is different from, the details of session persistence and context compaction.
Version prompts, tools and evidence so regressions are attributable
When a new model, system prompt, tool description, business rule or source collection is deployed, its impact should be compared against a stable control set. Save a run manifest containing model identifier, prompt revision, tool schema/version, policy revision, source fixture hash, test-case ID, grader version, timestamp and final status. For a tool-using agent, also retain a privacy-safe trace of the tool names, permitted inputs and summarized outcome. Without these dimensions, a changed success rate has too many possible causes.
Use representative holdout cases that were not repeatedly used to tune the prompt. If a team improves only the same visible examples, the model may learn their wording while remaining fragile to new records. Keep critical negative examples available as regression guards even when they are unpopular with a model’s average score. A new release should pass the same externally enforced authorization tests that the previous accepted release passed.
Compare meaningful rates, not just a single blended number. Success on policy retrieval, correct approval routing, transaction reconciliation, citation validity, improper cross-tenant access attempts, latency and human escalation are different observables. A weighted aggregate can help triage, but cannot erase an unauthorized payment. Record a "not tested" state when a required integration environment or source system was unavailable; a skipped test must not quietly count as success.
Put the release gate outside the model
A dependable release process has a deterministic condition: required suites run against an identifiable version, critical violations are zero, major regressions have disposition, and an authorized owner can see the remaining exceptions. A model may recommend a release, but the deployment system must inspect the test records and permissions itself. In a CI pipeline, keep secrets out of model-visible logs, run validators independently, and reserve production credentials for the narrow deployment step.
The same principle applies after deployment. Shadow evaluation can compare new responses without giving the candidate system permission to execute external writes. Canary traffic can expose latency, retrieval and routing regressions in bounded traffic. If a critical violation appears, stop the affected capability and return to a known accepted version. Human override and rollback mechanisms should be exercised in testing rather than described only in a runbook.
An engineer should be able to answer: Which failure made us withhold release? Which trace proves that no unauthorized action occurred? Which cases were not tested? What changed between the last accepted build and this one? What happens when evidence conflicts with the assistant’s proposed answer? Those questions turn evaluation from an attractive dashboard into a real safety and reliability mechanism.
A compact practical exercise
For a self-contained exercise, use a fictitious support policy, ten synthetic orders, a mocked read-only order service, a mocked refund service, and a simulated review queue. Specify three outcomes for each case: the allowed business state, the allowed tool actions and the essential customer-facing message. Execute only against mocks. In the lost-acknowledgement case, have the service commit a refund but return a timeout; then examine whether your orchestrator creates a duplicate or halts for reconciliation. In the contradictory-policy case, ensure an unsupported confident answer fails even if it reads well.
Introduce one deliberately deceptive evaluation pair: two agents give the same correct customer-facing answer, but the first respects the read-only tool policy while the second attempts an unauthorized refund before producing its reply. A response-only judge may give them identical grades; a trace-aware release gate must reject the second. Then reverse the test by keeping tool behavior safe but making the explanation omit a material policy conflict. Both failures matter, and neither can be repaired by adjusting a cosmetic score threshold. For an answer to independent Architect Foundations practice questions, require an executor-side check of the permitted business state and a separate explanation-quality check. A polished final message cannot compensate for an unauthorized refund attempt visible in the tool trace.
A release decision should rest on reproducible artifacts rather than an invented pass percentage: case identifiers, expected business states, tool traces, failure classifications, grading rationale, explicit stop conditions and an inventory of integrations that were not tested. Give a second engineer those artifacts and ask them to explain why the candidate build should ship or remain blocked. If the reviewer cannot reproduce the decision from the evidence, the test program is not yet a trustworthy release gate.