Question 470
A multi-tool AI agent books internal resources and queries enterprise information. The team is required to check that selected tool calls carry the right identifiers, dates, and authorized parameter values. Which option is the best fit?
- Tool argument correctness
- Task completion evaluator
- Tool failure recovery assessment
- Tool selection accuracy
Explanation
Correct Answer
A:Argument validation detects failures after tool choice but before execution. A correct tool name with a wrong customer ID can cause serious errors.
Incorrect Answers
B:Task completion examines end-to-end success, not merely a confident final message; an agent can claim success while an external operation failed; this approach leaves the team unable to check that selected tool calls carry the right identifiers, dates, and authorized parameter values.
C:Recovery testing checks how the agent reacts to dependency faults; hiding an error behind a plausible completion harms operational trust; the needed capability is to check that selected tool calls carry the right identifiers, dates, and authorized parameter values.
D:Tool selection evaluates planning at the action boundary; a well-worded final answer does not prove the correct tool was called; the requirement, by contrast, is to check that selected tool calls carry the right identifiers, dates, and authorized parameter values.
Question 471
A multi-tool AI agent books internal resources and queries enterprise information. An operator is tasked to assess whether the multi-step agent actually achieved the requested outcome according to verifiable state. Which mechanism provides that capability?
- Tool selection accuracy
- Tool permission boundary test
- Tool argument correctness
- Task completion evaluator
Explanation
Correct Answer
D:Task completion examines end-to-end success, not merely a confident final message. An agent can claim success while an external operation failed.
Incorrect Answers
A:Tool selection evaluates planning at the action boundary; a well-worded final answer does not prove the correct tool was called; the requirement, by contrast, is to assess whether the multi-step agent actually achieved the requested outcome according to verifiable state.
B:Authorization testing verifies external enforcement, not just the language model's willingness to refuse; a model-generated rationale is not a security control; the team would still need to assess whether the multi-step agent actually achieved the requested outcome according to verifiable state.
C:Argument validation detects failures after tool choice but before execution; a correct tool name with a wrong customer ID can cause serious errors; the specified outcome instead requires the team to assess whether the multi-step agent actually achieved the requested outcome according to verifiable state.
Question 472
An agent selects the correct billing tool but attempts to view another customer's private records using the current user's session. Which test verifies that tool execution respects the invoking user's authorization boundary?
- Task completion evaluator
- Tool selection accuracy
- Tool failure recovery assessment
- Tool permission boundary test
Explanation
Correct Answer
D:Authorization testing verifies external enforcement, not just the language model's willingness to refuse. A model-generated rationale is not a security control.
Incorrect Answers
A:Task completion examines end-to-end success, not merely a confident final message; an agent can claim success while an external operation failed; the implementation under review needs to ensure the agent cannot perform operations its invoking user is not authorized to request.
B:Tool selection evaluates planning at the action boundary; a well-worded final answer does not prove the correct tool was called; the requested task remains to ensure the agent cannot perform operations its invoking user is not authorized to request.
C:Recovery testing checks how the agent reacts to dependency faults; hiding an error behind a plausible completion harms operational trust; this alternative does not achieve the requirement to ensure the agent cannot perform operations its invoking user is not authorized to request.
Question 473
A multi-tool AI agent books internal resources and queries enterprise information. The team is required to generate representative multi-turn interactions to test state handling, clarifying questions, and task recovery. Which option is the best fit?
- Tool failure recovery assessment
- Tool permission boundary test
- Tool argument correctness
- Conversation simulation
Explanation
Correct Answer
D:Simulated conversations exercise long-lived agent behavior that single-turn prompts miss. Scenarios should still be checked for realism and target coverage.
Incorrect Answers
A:Recovery testing checks how the agent reacts to dependency faults; hiding an error behind a plausible completion harms operational trust; the actual design criterion is to generate representative multi-turn interactions to test state handling, clarifying questions, and task recovery.
B:Authorization testing verifies external enforcement, not just the language model's willingness to refuse; a model-generated rationale is not a security control; the outstanding need is to generate representative multi-turn interactions to test state handling, clarifying questions, and task recovery.
C:Argument validation detects failures after tool choice but before execution; a correct tool name with a wrong customer ID can cause serious errors; this approach leaves the team unable to generate representative multi-turn interactions to test state handling, clarifying questions, and task recovery.
Question 474
An internal booking agent selects the appropriate reservation tool but sometimes supplies an incorrect date. When that tool fails, the agent claims the booking succeeded instead of reporting the error or following its approved retry policy. Which TWO evaluator checks directly capture these defects? Choose TWO.
- Tool permission boundary test
- Conversation simulation
- Task completion evaluator
- Tool argument correctness
- Tool failure recovery assessment
Explanation
Correct Answers
D:Argument validation detects failures after tool choice but before execution; a correct tool name with a wrong customer ID can cause serious errors. Here, tool-call inputs are checked for the precise authorized date and identifiers.
E:Recovery testing checks how the agent reacts to dependency faults; hiding an error behind a plausible completion harms operational trust. Here, the evaluator penalizes fabricated success after a tool failure.
Incorrect Answers
A:Authorization testing verifies external enforcement, not just the language model's willingness to refuse; a model-generated rationale is not a security control; this mechanism leaves unresolved invalid tool parameters and false claims of success after a failing tool call.
B:Simulated conversations exercise long-lived agent behavior that single-turn prompts miss; scenarios should still be checked for realism and target coverage; it does not by itself address invalid tool parameters and false claims of success after a failing tool call.
C:Task completion examines end-to-end success, not merely a confident final message; an agent can claim success while an external operation failed; the observed fault still involves invalid tool parameters and false claims of success after a failing tool call.
Question 475
A business domain requires quality checks not fully covered by built-in model evaluators. The immediate design goal is to implement a domain-specific evaluator for mandatory financial disclaimers or regulated answer constraints. Which approach addresses that goal?
- Custom scoring rubric
- Evaluator version pinning
- Model-as-judge evaluator
- Evaluation disagreement review
Explanation
Correct Answer
A:A custom rubric expresses business rules that generic relevance cannot encode. Scoring logic and examples should be versioned and validated.
Incorrect Answers
B:Version pinning reduces measurement drift caused by changing the evaluator itself; otherwise score differences may reflect new judging rules; the implementation under review needs to implement a domain-specific evaluator for mandatory financial disclaimers or regulated answer constraints.
C:Model-based judging can scale subjective assessments but may introduce model-specific bias and cost; human calibration is valuable for critical decisions; the requested task remains to implement a domain-specific evaluator for mandatory financial disclaimers or regulated answer constraints.
D:Disagreement analysis identifies rubric ambiguity, flawed labels, or evaluator limitations; automatically trusting either side without investigation loses useful evidence; this alternative does not achieve the requirement to implement a domain-specific evaluator for mandatory financial disclaimers or regulated answer constraints.
Question 476
A business domain requires quality checks not fully covered by built-in model evaluators. The team is required to use an executable field or keyword constraint where the expected behavior can be checked exactly. Which option is the best fit?
- Evaluation disagreement review
- Custom scoring rubric
- Deterministic validation rule
- Evaluator version pinning
- Model-as-judge evaluator
Explanation
Correct Answer
C:Deterministic checks are stable for structural requirements. A generative judge may be unnecessary and less reproducible for simple schema constraints.
Incorrect Answers
A:Disagreement analysis identifies rubric ambiguity, flawed labels, or evaluator limitations; automatically trusting either side without investigation loses useful evidence; this alternative does not achieve the requirement to use an executable field or keyword constraint where the expected behavior can be checked exactly.
B:A custom rubric expresses business rules that generic relevance cannot encode; scoring logic and examples should be versioned and validated; the actual design criterion is to use an executable field or keyword constraint where the expected behavior can be checked exactly.
D:Version pinning reduces measurement drift caused by changing the evaluator itself; otherwise score differences may reflect new judging rules; this approach leaves the team unable to use an executable field or keyword constraint where the expected behavior can be checked exactly.
E:Model-based judging can scale subjective assessments but may introduce model-specific bias and cost; human calibration is valuable for critical decisions; the needed capability is to use an executable field or keyword constraint where the expected behavior can be checked exactly.
Question 477
A business domain requires quality checks not fully covered by built-in model evaluators. An operator is tasked to use an explicitly defined rubric and qualified judge model to score nuanced response attributes. Which mechanism provides that capability?
- Deterministic validation rule
- Evaluator calibration dataset
- Custom scoring rubric
- Model-as-judge evaluator
Explanation
Correct Answer
D:Model-based judging can scale subjective assessments but may introduce model-specific bias and cost. Human calibration is valuable for critical decisions.
Incorrect Answers
A:Deterministic checks are stable for structural requirements; a generative judge may be unnecessary and less reproducible for simple schema constraints; this approach leaves the team unable to use an explicitly defined rubric and qualified judge model to score nuanced response attributes.
B:Calibration reveals false positives and false negatives of a scoring method; an evaluator that produces numbers is not automatically a valid measure; the needed capability is to use an explicitly defined rubric and qualified judge model to score nuanced response attributes.
C:A custom rubric expresses business rules that generic relevance cannot encode; scoring logic and examples should be versioned and validated; the requirement, by contrast, is to use an explicitly defined rubric and qualified judge model to score nuanced response attributes.
Question 478
A business domain requires quality checks not fully covered by built-in model evaluators. The immediate design goal is to compare evaluator outcomes against expert-labeled examples before trusting it as a release gate. Which approach addresses that goal?
- Deterministic validation rule
- Model-as-judge evaluator
- Evaluator version pinning
- Evaluator calibration dataset
Explanation
Correct Answer
D:Calibration reveals false positives and false negatives of a scoring method. An evaluator that produces numbers is not automatically a valid measure.
Incorrect Answers
A:Deterministic checks are stable for structural requirements; a generative judge may be unnecessary and less reproducible for simple schema constraints; the team would still need to compare evaluator outcomes against expert-labeled examples before trusting it as a release gate.
B:Model-based judging can scale subjective assessments but may introduce model-specific bias and cost; human calibration is valuable for critical decisions; the specified outcome instead requires the team to compare evaluator outcomes against expert-labeled examples before trusting it as a release gate.
C:Version pinning reduces measurement drift caused by changing the evaluator itself; otherwise score differences may reflect new judging rules; the implementation under review needs to compare evaluator outcomes against expert-labeled examples before trusting it as a release gate.
Question 479
A business domain requires quality checks not fully covered by built-in model evaluators. The team is required to preserve the exact rubric, implementation, and judge configuration used to compare successive releases. Which option is the best fit?
- Evaluator version pinning
- Evaluator calibration dataset
- Evaluation disagreement review
- Deterministic validation rule
Explanation
Correct Answer
A:Version pinning reduces measurement drift caused by changing the evaluator itself. Otherwise score differences may reflect new judging rules.
Incorrect Answers
B:Calibration reveals false positives and false negatives of a scoring method; an evaluator that produces numbers is not automatically a valid measure; this alternative does not achieve the requirement to preserve the exact rubric, implementation, and judge configuration used to compare successive releases.
C:Disagreement analysis identifies rubric ambiguity, flawed labels, or evaluator limitations; automatically trusting either side without investigation loses useful evidence; the actual design criterion is to preserve the exact rubric, implementation, and judge configuration used to compare successive releases.
D:Deterministic checks are stable for structural requirements; a generative judge may be unnecessary and less reproducible for simple schema constraints; the outstanding need is to preserve the exact rubric, implementation, and judge configuration used to compare successive releases.
Question 480
A business domain requires quality checks not fully covered by built-in model evaluators. An operator is tasked to investigate cases where human reviewers and automated evaluators reach substantially different conclusions. Which mechanism provides that capability?
- Evaluator version pinning
- Deterministic validation rule
- Evaluation disagreement review
- Model-as-judge evaluator
Explanation
Correct Answer
C:Disagreement analysis identifies rubric ambiguity, flawed labels, or evaluator limitations. Automatically trusting either side without investigation loses useful evidence.
Incorrect Answers
A:Version pinning reduces measurement drift caused by changing the evaluator itself; otherwise score differences may reflect new judging rules; the outstanding need is to investigate cases where human reviewers and automated evaluators reach substantially different conclusions.
B:Deterministic checks are stable for structural requirements; a generative judge may be unnecessary and less reproducible for simple schema constraints; this approach leaves the team unable to investigate cases where human reviewers and automated evaluators reach substantially different conclusions.
D:Model-based judging can scale subjective assessments but may introduce model-specific bias and cost; human calibration is valuable for critical decisions; the requirement, by contrast, is to investigate cases where human reviewers and automated evaluators reach substantially different conclusions.