Question 220
An evaluation's expected answers have no source records or revision dates. Several policies have changed. What is needed before trusting failures against that set?
- Replace every expected answer with the candidate's response
- Verify references against authoritative versions and record provenance
- Assume the oldest answer key is always authoritative
- Ignore all failures involving policy questions
Explanation
Correct Answer
B:A model can disagree with an outdated answer while following current policy. Evaluation truth needs its own maintained evidence and version history.
Incorrect Answers
A:That makes the model its own answer key. It removes the independent standard the evaluation is supposed to apply.
C:Age does not establish current correctness. Reference material must match the intended task version and applicable policy period.
D:Excluding the relevant domain avoids the verification problem by abandoning coverage. The reference labels should be repaired instead.
Question 221
An eligibility assistant should change its result when the documented income threshold is crossed, while other facts remain constant. Which test best checks that behavior?
- Repeated copies of a single qualifying case
- Unrelated cases from different industries
- A readability score for the explanation
- Paired cases differing only around the decisive threshold
Explanation
Correct Answer
D:Controlled pairs isolate whether the governing variable changes the decision appropriately. They reveal failures that broad aggregate accuracy can miss.
Incorrect Answers
A:Repetition can measure consistency but does not test the boundary. Both sides of the threshold need representation.
B:Many changing factors make the cause of different answers unclear. The desired test isolates one decision-relevant variable.
C:Readable prose does not establish that the eligibility rule responds correctly. The test must evaluate the actual decision under changed inputs.
Question 222
Reordering independent items in an invoice should not change their total. Which evaluation can detect an ordering-dependent calculation error?
- Compare invoices with different amounts and currencies
- Ask the model whether ordering should matter
- Compare outputs before and after a meaning-preserving reorder
- Score only whether both answers are grammatical
Explanation
Correct Answer
C:A metamorphic test checks a known relationship between inputs and outputs. Here the transformation changes presentation while preserving the required aggregate.
Incorrect Answers
A:Those changes legitimately alter the result, so they do not isolate an ordering defect. The transformation should preserve the expected total.
B:A statement of the rule does not demonstrate compliant execution. The evaluation should run both transformed cases and compare their outcomes.
D:Grammar does not verify the invariant. Two fluent responses can report different totals for equivalent item sets.
Question 223
A RAG evaluation contains only questions with clear answers in the corpus. What important behavior remains untested?
- Handling questions for which approved evidence is absent
- Answer quality for questions whose supporting passage ranks near the retrieval cutoff
- Correctness on questions answered by more than one supplied passage
- Consistency when two answerable questions use different wording
Explanation
Correct Answer
A:A complete evaluation should test whether the system recognizes unsupported requests. Answerable-only cases cannot reveal fabricated answers to missing information.
Incorrect Answers
B:Ranking difficulty can occur even when an answer exists. The all-answerable corpus does not specifically exclude such cases; the definite missing behavior is handling absent evidence.
C:Multi-passage support can be present in an all-answerable set. The stated construction does not establish that this behavior is missing.
D:Paraphrases can both have clear corpus answers. Their inclusion is independent of the missing no-evidence cases identified in the stem.
Question 224
A new guardrail reduces wrong answers by refusing nearly every question. Which evaluation prevents calling this a useful improvement?
- Remove easy answerable questions from the evaluation
- Report only the number of incorrect generated claims
- Measure correctness together with answer coverage and refusal appropriateness
- Count every refusal as a successful answer
Explanation
Correct Answer
C:Low error among answered cases can hide excessive refusal. The service must remain useful on supported requests while avoiding unsupported conclusions.
Incorrect Answers
A:Removing supported cases hides the usability loss. The evaluation needs both answerable and unanswerable requests to assess the trade-off.
B:That metric can improve when the system stops answering entirely. It does not capture whether legitimate user needs are being met.
D:Refusal is correct only under appropriate conditions. Treating all refusals as success rewards avoidance rather than task performance.
Question 225
An answer cites a real document that discusses the topic but does not support the stated conclusion. What evaluation dimension catches this?
- Whether the document URL can be opened
- Whether the source title resembles the question
- Whether every paragraph contains a citation marker
- Whether the cited passage entails or supports the claim
Explanation
Correct Answer
D:Citation existence and topical relevance are insufficient. The evaluator must check the relationship between the specific statement and its cited evidence.
Incorrect Answers
A:Accessibility proves the reference exists, not that it supports the conclusion. A reachable source can still be misused.
B:Topic similarity is weaker than claim support. The conclusion may go beyond what a related document actually establishes.
C:Marker frequency measures formatting. It can reward unsupported claims decorated with valid-looking references.
Question 226
A summary accurately states two of five required findings and omits the rest. Which rubric design identifies the failure?
- Use answer length as a complete substitute
- Accept any summary containing the main topic
- Score only the truth of statements that appear
- Score required finding coverage separately from factual correctness
Explanation
Correct Answer
D:An answer can contain no false statements and still be incomplete. Separate coverage checks ensure all required components are represented.
Incorrect Answers
A:A long answer can repeat two findings without adding the missing three. Length does not establish semantic coverage.
B:Topic recognition is much weaker than meeting the requested content contract. The rubric must represent the actual required findings.
C:This ignores omissions and can reward very short partial answers. The task requires both accuracy and coverage of specified findings.
Question 227
A retriever returns ten passages, of which four are relevant to the question. What is precision for that returned set?
- The value cannot be computed without corpus size
- 25%
- 60%
- 40%
Explanation
Correct Answer
D:Precision is the fraction of retrieved items that are relevant: four divided by ten. It does not require counting relevant items never retrieved.
Incorrect Answers
A:Corpus size is not needed for precision on a returned set. It would matter to other measures or sampling questions, not this ratio.
B:This reverses the relevant and returned counts. The denominator for precision is all retrieved passages, not just the relevant subset.
C:Six of ten passages are irrelevant, so this is the complement of precision. It describes noise in the returned set.
Question 228
A labeled corpus contains eight passages relevant to a query. The retriever returns three of them, along with two irrelevant passages. What is recall?
- 62.5%
- 37.5%
- 100% because at least one relevant passage was found
- 60%
Explanation
Correct Answer
B:Recall measures how much relevant evidence was found: three of eight. The two irrelevant returns affect precision, not this denominator.
Incorrect Answers
A:Five of eight relevant passages were missed, so this is the missed-relevance fraction. Recall counts the relevant passages actually retrieved.
C:Finding some support does not establish complete coverage. Recall distinguishes partial retrieval from finding all labeled relevant evidence.
D:Three relevant passages out of five returned is precision. It answers a different question from coverage of all eight relevant passages.
Question 229
A candidate improves average quality but violates a mandatory prohibition on unauthorized writes in one test. How should the release gate treat that result?
- Fail the hard constraint regardless of the weighted average
- Reduce the safety weight until the candidate passes
- Approve because the weighted score increased
- Remove the violating case from the dataset
Explanation
Correct Answer
A:A mandatory boundary cannot be traded away through unrelated quality gains. The release decision must distinguish non-negotiable requirements from optimizable metrics.
Incorrect Answers
B:Changing weights after observing failure undermines the gate. It also misrepresents a hard constraint as an optional preference.
C:Averages can hide a critical violation. The prohibition was specified as mandatory, so other improvements do not compensate for breaking it.
D:Excluding a valid failure changes the test to favor the candidate. The defect should be corrected and the case retained.
Question 230
A comparison judge tends to prefer the first of two answers, even when their quality is similar. What evaluation control is useful?
- Keep one fixed ordering and average more judgments in that ordering
- Increase the maximum length of the judge's explanation
- Always put the new candidate first
- Randomize answer order and check order sensitivity
Explanation
Correct Answer
D:Order randomization reduces systematic positional advantage. Reversed-pair checks help measure whether the judge's preference follows quality or presentation position.
Incorrect Answers
A:Repeated scoring can reduce some variability while preserving a systematic position preference. Order sensitivity requires changing the ordering under controlled comparison.
B:More explanation does not necessarily remove positional bias. The experimental control must address the order in which candidates are presented.
C:That gives the candidate the known advantage and biases the comparison. It does not estimate genuine quality improvement.