Question 230
A comparison judge tends to prefer the first of two answers, even when their quality is similar. What evaluation control is useful?
- Keep one fixed ordering and average more judgments in that ordering
- Increase the maximum length of the judge's explanation
- Always put the new candidate first
- Randomize answer order and check order sensitivity
Explanation
Correct Answer
D:Order randomization reduces systematic positional advantage. Reversed-pair checks help measure whether the judge's preference follows quality or presentation position.
Incorrect Answers
A:Repeated scoring can reduce some variability while preserving a systematic position preference. Order sensitivity requires changing the ordering under controlled comparison.
B:More explanation does not necessarily remove positional bias. The experimental control must address the order in which candidates are presented.
C:That gives the candidate the known advantage and biases the comparison. It does not estimate genuine quality improvement.
Question 231
A judge rewards longer answers even when they repeat facts and omit required constraints. What should the rubric emphasize?
- Automatic rejection of every concise answer
- A minimum word count as the only quality measure
- A preference for whichever answer has more headings
- Task-specific correctness and coverage without rewarding length itself
Explanation
Correct Answer
D:The judge needs criteria tied to the actual output contract. Verbosity should not substitute for complete, accurate treatment of required conditions.
Incorrect Answers
A:Concise responses can fully satisfy the task. The evaluation should assess their content rather than impose an unrelated stylistic penalty.
B:Word count can be satisfied through repetition. It does not establish whether the answer includes the necessary constraints or avoids false claims.
C:Presentation can aid readability but is not the substantive objective. More headings may still organize an incomplete or incorrect answer.
Question 232
An automated grader disagrees with expert reviewers on consequential cases. What should precede using its score for automatic release decisions?
- Rescale all judge scores to the human score range without checking agreement
- Increase evaluation volume without reviewing disagreement causes
- Assume the grader is correct because it runs consistently
- Calibrate the rubric and grader against adjudicated expert examples
Explanation
Correct Answer
D:The grader's outputs need evidence of agreement with the intended standard. Disagreements should be analyzed before its scores become authoritative gates.
Incorrect Answers
A:Matching numeric ranges does not show that the judge ranks or explains cases like qualified reviewers. Calibration needs comparison against adjudicated examples.
B:More biased grades can make estimates more precise around the wrong target. The scoring mechanism must first reflect the intended standard.
C:Consistency can reproduce a systematic error. Repeated agreement with itself does not establish alignment with the task's true evaluation criteria.
Question 233
Historical evaluation scores rise after the judge model changes, while candidate outputs remain identical. What should be concluded?
- The measurement system changed and needs comparability checks
- The candidate necessarily improved despite identical outputs
- All older evaluations must be deleted
- Judge versions do not matter when the rubric text is fixed
Explanation
Correct Answer
A:Score movement can come from the grader rather than the product. Versioned judges and bridge evaluations help distinguish measurement drift from real improvement.
Incorrect Answers
B:Unchanged outputs provide no evidence of a product change. The known intervention occurred in how those outputs were scored.
C:Historical results remain useful if their measurement conditions are recorded. They should be interpreted with the judge-version difference in mind.
D:The same rubric can be interpreted differently by another model. Measurement behavior depends on both the rubric and the grader implementation.
Question 234
An agent succeeds once in ten trials but fails the other nine. The team reports that the task is solved. What additional result is necessary?
- The per-trial success rate and distribution of failure modes
- Only the transcript of the successful attempt
- The average response length across the ten trials
- The probability of success conditional on selecting only successful transcripts
Explanation
Correct Answer
A:One successful run establishes possibility, not reliability. Production decisions need the frequency and nature of failures under comparable conditions.
Incorrect Answers
B:Selecting the best run hides the dominant outcome. It cannot establish whether a typical user will receive a correct result.
C:Length does not measure whether the task succeeded reliably. Reporting it would leave the nine failed outcomes unaccounted for in the success claim.
D:Conditioning on success removes the failures that determine reliability. The denominator must include attempted trials, not just selected successful examples.
Question 235
A benchmark reports whether any of five attempts succeeds. Production allows only one attempt per request. What caution applies?
- The production budget is irrelevant to metric selection
- The benchmark proves every attempt succeeded
- Best-of-five success does not equal single-attempt reliability
- Five-attempt success is always lower than single-attempt success
Explanation
Correct Answer
C:Multiple chances can improve the probability of at least one success. A one-attempt service needs metrics matching its actual execution budget.
Incorrect Answers
A:Metrics should reflect the deployed opportunity to recover or retry. A benchmark with more chances can overstate the user-facing success rate.
B:An any-success criterion can pass even when four attempts fail. It does not describe consistency across all attempts.
D:Allowing additional attempts cannot be interpreted as inherently reducing the chance of at least one success under the same trial conditions.
Question 236
An agent test modifies a database, and later tests reuse that altered state. Scores depend on execution order. What should the harness provide?
- Random order without any state management
- Controlled reset or isolated state for independent test cases
- A larger model for cases executed last
- Deletion of every test that follows a failure
Explanation
Correct Answer
B:State leakage can make tests easier or harder for accidental reasons. Isolation makes outcomes attributable to each case's defined initial conditions.
Incorrect Answers
A:Randomization can reveal instability but does not eliminate contamination. The harness still needs a controlled state contract.
C:Changing the model masks the underlying test dependency. It makes comparisons less controlled rather than fixing the shared-state issue.
D:Removing affected cases hides the contamination. The correct remedy is a reliable setup and teardown process for the test environment.
Question 237
An agent says it updated a ticket, but the database still shows the old value. What should the evaluator grade for task completion?
- The authorized final state of the target system
- The syntactic validity of the generated update arguments
- Whether the agent invoked the update tool at least once
- Whether the transport accepted the update request
Explanation
Correct Answer
A:A completion claim is not proof that the operation succeeded. Evaluation should inspect the relevant observable state or trusted execution receipt.
Incorrect Answers
B:Valid arguments are only a prerequisite for a successful update. They do not prove that the operation committed the required final state.
C:An invocation can fail, be rejected, or target the wrong record. Completion requires checking the authorized outcome in the target system.
D:Transport acceptance need not imply a committed business operation. The unchanged database state contradicts a claim of completed update.
Question 238
An agent reaches the correct answer but reads a prohibited database along the way. Which evaluation is needed in addition to final-answer scoring?
- Check the execution trajectory against access and process constraints
- Ignore the path whenever the answer is correct
- Reward the agent for using more data sources
- Evaluate only the final answer's grammar
Explanation
Correct Answer
A:Correct output does not excuse an unauthorized path. The harness should inspect relevant tool use and policy adherence during execution.
Incorrect Answers
B:That treats consequential process constraints as irrelevant. The task requires both the right result and permitted means of obtaining it.
C:More sources can include unauthorized ones. Source count is not a substitute for checking whether each access was allowed.
D:Grammar does not reveal prohibited tool calls. The failure exists in execution behavior that output-only assessment can miss.
Question 239
A latency test uses short requests with no tools, but production includes long retrieval chains. What must improve before trusting the capacity estimate?
- Remove external dependencies from every performance test
- Use the fastest observed request as the capacity benchmark
- Represent production request mix and dependency behavior
- Increase only the number of easy requests
Explanation
Correct Answer
C:Load performance depends on the work each request performs. A lightweight synthetic path can substantially overstate capacity for the real workload.
Incorrect Answers
A:Isolation tests are useful for components, but end-to-end capacity still requires evidence about the dependencies used in production.
B:The best case does not establish sustainable throughput or tail behavior. Capacity planning needs representative distributions under load.
D:More easy traffic stresses a different workload. It does not capture the costs of retrieval, long contexts, or external calls.
Question 240
Throughput stops increasing as concurrency rises, while queue delay grows sharply. What is the likely interpretation?
- A bottleneck has reached capacity and extra concurrency adds waiting
- The system needs longer queues as its only fix
- The model's answers are necessarily becoming less grammatical
- Every additional request is improving processing efficiency
Explanation
Correct Answer
A:Once a limiting resource saturates, more arrivals can increase latency without increasing completed work. Measurement should identify the constrained stage.
Incorrect Answers
B:A larger queue can absorb a temporary burst but cannot increase sustained service capacity. It may simply permit longer waiting times.
C:The observed metrics concern capacity, not language quality. Content changes would require separate evidence rather than inference from queue growth.
D:The flat throughput contradicts that claim. More concurrent work is accumulating as delay rather than being completed faster.