Question 560
A search-quality review finds that only four passages in the top ten retrieval results are marked relevant. Which metric reports the fraction of retrieved top-k candidates that are relevant, rather than the proportion of all relevant passages recovered?
- End-to-end answer success rate
- Retrieval precision at k
- Retrieval recall at k
- Grounded answer scoring
- Retrieval miss categorization
Explanation
Correct Answer
B:Precision at k captures ranking noise within the evidence budget. It can decline as more marginal results are included.
Incorrect Answers
A:End-to-end success combines evidence discovery and answer construction; a good individual search metric may not translate into an acceptable user outcome; the outstanding need is to measure the fraction of the top k candidates that are labeled relevant to the query.
C:Recall at k tests evidence discovery; it does not judge whether the language model uses that evidence faithfully; the needed capability is to measure the fraction of the top k candidates that are labeled relevant to the query.
D:Groundedness evaluates answer synthesis against evidence; strong retrieval recall does not guarantee the model will avoid unsupported assertions; the requirement, by contrast, is to measure the fraction of the top k candidates that are labeled relevant to the query.
E:Categorization links symptoms to the stage responsible for absent evidence; prompt rewriting alone cannot recover unindexed documents; the team would still need to measure the fraction of the top k candidates that are labeled relevant to the query.
Question 561
A search team must separate retrieval failure from generation failure in a low-quality assistant. An operator is tasked to reward systems that place the first relevant passage near the top of the returned list. Which mechanism provides that capability?
- Retrieval recall at k
- Retrieval miss categorization
- Mean reciprocal rank
- End-to-end answer success rate
Explanation
Correct Answer
C:Mean reciprocal rank emphasizes the rank of the earliest correct hit. It does not measure how many other relevant passages were found.
Incorrect Answers
A:Recall at k tests evidence discovery; it does not judge whether the language model uses that evidence faithfully; the requirement, by contrast, is to reward systems that place the first relevant passage near the top of the returned list.
B:Categorization links symptoms to the stage responsible for absent evidence; prompt rewriting alone cannot recover unindexed documents; the team would still need to reward systems that place the first relevant passage near the top of the returned list.
D:End-to-end success combines evidence discovery and answer construction; a good individual search metric may not translate into an acceptable user outcome; the implementation under review needs to reward systems that place the first relevant passage near the top of the returned list.
Question 562
A retrieval trace contains the correct source document, but the generated answer still makes a claim that the source does not support. Which evaluator focuses on evidentiary support of the final answer rather than retrieval position?
- End-to-end answer success rate
- Grounded answer scoring
- Retrieval recall at k
- Mean reciprocal rank
Explanation
Correct Answer
B:Groundedness evaluates answer synthesis against evidence. Strong retrieval recall does not guarantee the model will avoid unsupported assertions.
Incorrect Answers
A:End-to-end success combines evidence discovery and answer construction; a good individual search metric may not translate into an acceptable user outcome; the implementation under review needs to check whether the final generated claims are supported by the actual retrieved context.
C:Recall at k tests evidence discovery; it does not judge whether the language model uses that evidence faithfully; this alternative does not achieve the requirement to check whether the final generated claims are supported by the actual retrieved context.
D:Mean reciprocal rank emphasizes the rank of the earliest correct hit; it does not measure how many other relevant passages were found; the actual design criterion is to check whether the final generated claims are supported by the actual retrieved context.
Question 563
A search team must separate retrieval failure from generation failure in a low-quality assistant. The team is required to tag failures as indexing, chunking, embedding, filtering, or ranking problems before choosing remediation. Which option is the best fit?
- Retrieval miss categorization
- Retrieval precision at k
- End-to-end answer success rate
- Retrieval recall at k
Explanation
Correct Answer
A:Categorization links symptoms to the stage responsible for absent evidence. Prompt rewriting alone cannot recover unindexed documents.
Incorrect Answers
B:Precision at k captures ranking noise within the evidence budget; it can decline as more marginal results are included; the outstanding need is to tag failures as indexing, chunking, embedding, filtering, or ranking problems before choosing remediation.
C:End-to-end success combines evidence discovery and answer construction; a good individual search metric may not translate into an acceptable user outcome; this approach leaves the team unable to tag failures as indexing, chunking, embedding, filtering, or ranking problems before choosing remediation.
D:Recall at k tests evidence discovery; it does not judge whether the language model uses that evidence faithfully; the needed capability is to tag failures as indexing, chunking, embedding, filtering, or ranking problems before choosing remediation.
Question 564
The team has relevance labels for retrieved passages and verified end-user outcomes for its RAG assistant. It needs to distinguish the proportion of irrelevant items in the top-k retrieved set from the fraction of complete requests the full assistant handles correctly. Which TWO metrics answer those separate questions? Choose TWO.
- Retrieval recall at k
- Retrieval precision at k
- End-to-end answer success rate
- Grounded answer scoring
- Mean reciprocal rank
Explanation
Correct Answers
B:Precision at k captures ranking noise within the evidence budget; it can decline as more marginal results are included. Here, the retrieved candidate list is scored for the fraction that is relevant.
C:End-to-end success combines evidence discovery and answer construction; a good individual search metric may not translate into an acceptable user outcome. Here, the full retrieval-plus-generation workflow is judged against successful user outcomes.
Incorrect Answers
A:Recall at k tests evidence discovery; it does not judge whether the language model uses that evidence faithfully; this mechanism leaves unresolved retrieval-list precision and validated full-task success.
D:Groundedness evaluates answer synthesis against evidence; strong retrieval recall does not guarantee the model will avoid unsupported assertions; the immediate requirement instead concerns retrieval-list precision and validated full-task success.
E:Mean reciprocal rank emphasizes the rank of the earliest correct hit; it does not measure how many other relevant passages were found; the system would still need a remedy for retrieval-list precision and validated full-task success.
Question 565
An application team evaluates competing retrieval improvements before production rollout. The immediate design goal is to keep a fixed representative set of user queries to compare retrieval configurations fairly. Which approach addresses that goal?
- Query-set holdout
- Chunking A/B experiment
- Production cohort rollout
- Reranker experiment
Explanation
Correct Answer
A:A stable query set isolates the effects of indexing and search changes. Changing the benchmark during comparison can invalidate conclusions.
Incorrect Answers
B:Controlled chunking trials show whether boundaries affect retrieval and answer quality; changing the generator simultaneously would confound attribution; the implementation under review needs to keep a fixed representative set of user queries to compare retrieval configurations fairly.
C:Cohort rollout limits exposure to ranking changes; it should have clear assignment, telemetry, and rollback criteria; the requested task remains to keep a fixed representative set of user queries to compare retrieval configurations fairly.
D:Reranker comparisons identify quality gains and latency costs; results require stable judgment labels and retrieval settings; this alternative does not achieve the requirement to keep a fixed representative set of user queries to compare retrieval configurations fairly.
Question 566
An application team evaluates competing retrieval improvements before production rollout. The team is required to compare two chunk sizes while holding model version and scoring criteria constant. Which option is the best fit?
- Latency-quality Pareto review
- Chunking A/B experiment
- Reranker experiment
- Hybrid versus vector A/B
Explanation
Correct Answer
B:Controlled chunking trials show whether boundaries affect retrieval and answer quality. Changing the generator simultaneously would confound attribution.
Incorrect Answers
A:A Pareto review identifies useful tradeoffs among quality, cost, and speed; selecting only maximum recall may create unacceptable response time; this alternative does not achieve the requirement to compare two chunk sizes while holding model version and scoring criteria constant.
C:Reranker comparisons identify quality gains and latency costs; results require stable judgment labels and retrieval settings; the outstanding need is to compare two chunk sizes while holding model version and scoring criteria constant.
D:Paired search experiments capture exact-code and paraphrase behavior; a theoretical advantage should be checked on actual queries; this approach leaves the team unable to compare two chunk sizes while holding model version and scoring criteria constant.
Question 567
An application team evaluates competing retrieval improvements before production rollout. An operator is tasked to measure whether combining lexical and vector retrieval improves results for mixed technical query types. Which mechanism provides that capability?
- Query-set holdout
- Reranker experiment
- Latency-quality Pareto review
- Production cohort rollout
- Hybrid versus vector A/B
Explanation
Correct Answer
E:Paired search experiments capture exact-code and paraphrase behavior. A theoretical advantage should be checked on actual queries.
Incorrect Answers
A:A stable query set isolates the effects of indexing and search changes; changing the benchmark during comparison can invalidate conclusions; this approach leaves the team unable to measure whether combining lexical and vector retrieval improves results for mixed technical query types.
B:Reranker comparisons identify quality gains and latency costs; results require stable judgment labels and retrieval settings; the needed capability is to measure whether combining lexical and vector retrieval improves results for mixed technical query types.
C:A Pareto review identifies useful tradeoffs among quality, cost, and speed; selecting only maximum recall may create unacceptable response time; the requirement, by contrast, is to measure whether combining lexical and vector retrieval improves results for mixed technical query types.
D:Cohort rollout limits exposure to ranking changes; it should have clear assignment, telemetry, and rollback criteria; the team would still need to measure whether combining lexical and vector retrieval improves results for mixed technical query types.
Question 568
An application team evaluates competing retrieval improvements before production rollout. The immediate design goal is to test a second-stage semantic ranker against a no-rerank baseline using the same initial candidate rules. Which approach addresses that goal?
- Reranker experiment
- Production cohort rollout
- Chunking A/B experiment
- Hybrid versus vector A/B
Explanation
Correct Answer
A:Reranker comparisons identify quality gains and latency costs. Results require stable judgment labels and retrieval settings.
Incorrect Answers
B:Cohort rollout limits exposure to ranking changes; it should have clear assignment, telemetry, and rollback criteria; the specified outcome instead requires the team to test a second-stage semantic ranker against a no-rerank baseline using the same initial candidate rules.
C:Controlled chunking trials show whether boundaries affect retrieval and answer quality; changing the generator simultaneously would confound attribution; the implementation under review needs to test a second-stage semantic ranker against a no-rerank baseline using the same initial candidate rules.
D:Paired search experiments capture exact-code and paraphrase behavior; a theoretical advantage should be checked on actual queries; the requested task remains to test a second-stage semantic ranker against a no-rerank baseline using the same initial candidate rules.
Question 569
An application team evaluates competing retrieval improvements before production rollout. The team is required to choose a retrieval configuration that meets quality targets without violating service latency budgets. Which option is the best fit?
- Reranker experiment
- Latency-quality Pareto review
- Production cohort rollout
- Query-set holdout
Explanation
Correct Answer
B:A Pareto review identifies useful tradeoffs among quality, cost, and speed. Selecting only maximum recall may create unacceptable response time.
Incorrect Answers
A:Reranker comparisons identify quality gains and latency costs; results require stable judgment labels and retrieval settings; the requested task remains to choose a retrieval configuration that meets quality targets without violating service latency budgets.
C:Cohort rollout limits exposure to ranking changes; it should have clear assignment, telemetry, and rollback criteria; the actual design criterion is to choose a retrieval configuration that meets quality targets without violating service latency budgets.
D:A stable query set isolates the effects of indexing and search changes; changing the benchmark during comparison can invalidate conclusions; the outstanding need is to choose a retrieval configuration that meets quality targets without violating service latency budgets.
Question 570
An application team evaluates competing retrieval improvements before production rollout. An operator is tasked to route a limited share of users to an approved retrieval variant and monitor real-world task success before expansion. Which mechanism provides that capability?
- Query-set holdout
- Chunking A/B experiment
- Hybrid versus vector A/B
- Production cohort rollout
Explanation
Correct Answer
D:Cohort rollout limits exposure to ranking changes. It should have clear assignment, telemetry, and rollback criteria.
Incorrect Answers
A:A stable query set isolates the effects of indexing and search changes; changing the benchmark during comparison can invalidate conclusions; the outstanding need is to route a limited share of users to an approved retrieval variant and monitor real-world task success before expansion.
B:Controlled chunking trials show whether boundaries affect retrieval and answer quality; changing the generator simultaneously would confound attribution; this approach leaves the team unable to route a limited share of users to an approved retrieval variant and monitor real-world task success before expansion.
C:Paired search experiments capture exact-code and paraphrase behavior; a theoretical advantage should be checked on actual queries; the needed capability is to route a limited share of users to an approved retrieval variant and monitor real-world task success before expansion.