Question 1
After implementing a workflow to store approved long-term user preferences separately, the team needs a robust release gate. Which check should it choose?
- Test multi-turn follow-ups for correct reference resolution without cross-user leakage
- Verify preference recall, deletion behavior, and strict user-by-user isolation
- Trace tool invocations and compare returned statuses with authorized API ground truth
- Compare retrieval success on mixed identifier-and-paraphrase evaluation queries
Explanation
Correct Answer
B:The accepted change is to store approved long-term user preferences separately; this evaluation checks whether it can personalize repeated interactions without placing all old transcripts in every prompt, since explicitly persisted, consent-governed preference data supports bounded long-term personalization; transient chat state should not become an uncontrolled permanent profile.
Incorrect Answers
A:This test concerns the ability to resolve a follow-up such as 'What about the second option?' using the user's previous turn, for which conversation state preserves necessary prior turns for reference resolution; it differs from a searchable repository of long-lived institutional documents; it does not establish whether the change to store approved long-term user preferences separately works for the current release.
C:This test concerns the ability to let an agent retrieve a live order status from the organization's transactional system, for which an authenticated tool call obtains current application state at request time; stale embeddings cannot guarantee a transaction's latest status; it does not establish whether the change to store approved long-term user preferences separately works for the current release.
D:This test concerns the ability to support queries mixing specific document codes with natural-language intent, for which hybrid retrieval runs lexical and vector queries together and merges ranked results; it helps preserve exact tokens while matching semantic concepts; it does not establish whether the change to store approved long-term user preferences separately works for the current release.
Question 2
A release reviewer is assessing a feature intended to ground responses in a search knowledge source. Which test is most meaningful?
- Trace tool invocations and compare returned statuses with authorized API ground truth
- Test multi-turn follow-ups for correct reference resolution without cross-user leakage
- Score generated functions on passing unit tests and absence of unsafe execution behavior
- Audit answer support against current source passages and check citation correctness
Explanation
Correct Answer
D:The accepted change is to ground responses in a search knowledge source; this evaluation checks whether it can answer policy questions using the latest approved internal manuals instead of model recollection, since a connected retrieval source gives an agent current authorized reference content that can be cited and updated separately from model weights.
Incorrect Answers
A:This test concerns the ability to let an agent retrieve a live order status from the organization's transactional system, for which an authenticated tool call obtains current application state at request time; stale embeddings cannot guarantee a transaction's latest status; it does not establish whether the change to ground responses in a search knowledge source works for the current release.
B:This test concerns the ability to resolve a follow-up such as 'What about the second option?' using the user's previous turn, for which conversation state preserves necessary prior turns for reference resolution; it differs from a searchable repository of long-lived institutional documents; it does not establish whether the change to ground responses in a search knowledge source works for the current release.
C:This test concerns the ability to produce and review Python functions with useful syntax and unit-test coverage, for which a code-capable generative model can reason over programming context; repository access, tool execution, and tests still need explicit integration; it does not establish whether the change to ground responses in a search knowledge source works for the current release.
Question 3
An operator can reproduce this error: A general-purpose large model passes accuracy checks but exceeds the response-time budget for short tickets. Which targeted repair is justified?
- Pass the actual image to a vision-capable model and ground its response in visible evidence
- Enable semantic ranking over searchable human-readable fields after initial retrieval
- Embed the queries and chunks with a compatible model and add vector retrieval
- Benchmark a smaller model on labeled tickets and route this bounded task to it
Explanation
Correct Answer
D:The fault signal is that a general-purpose large model passes accuracy checks but exceeds the response-time budget for short tickets; the targeted fix works because a compact model can meet narrow classification needs with fewer inference resources when validated against representative labeled examples, restoring the ability to categorize short support tickets under strict response-time and inference-budget limits.
Incorrect Answers
A:If the failing function were to answer natural-language questions about photographed equipment where the answer depends on visible details, this change could be relevant because a multimodal model accepts image evidence alongside text; a text-only model cannot inspect visual input unless a separate vision pipeline supplies it; the actual symptom is that a general-purpose large model passes accuracy checks but exceeds the response-time budget for short tickets, which calls for a different repair.
B:If the failing function were to improve ordering of an already retrieved set by how directly it answers a user's question, this change could be relevant because semantic reranking uses language understanding to reorder candidate search results; it does not replace indexing or generate new source documents; the actual symptom is that a general-purpose large model passes accuracy checks but exceeds the response-time budget for short tickets, which calls for a different repair.
C:If the failing function were to map queries and reference passages into vectors for semantic retrieval, this change could be relevant because embeddings encode text into vectors used by a similarity search system; they are not themselves a response-generating chatbot; the actual symptom is that a general-purpose large model passes accuracy checks but exceeds the response-time budget for short tickets, which calls for a different repair.
Question 4
In an architecture review, the key requirement is to let an agent retrieve a live order status from the organization's transactional system. Which solution directly satisfies it?
- Expose an authenticated business API as an agent tool
- Store approved long-term user preferences separately
- Use conversation state to retain dialogue context
- Combine keyword and vectors in hybrid search
Explanation
Correct Answer
A:For the stated requirement to let an agent retrieve a live order status from the organization's transactional system, an authenticated tool call obtains current application state at request time; stale embeddings cannot guarantee a transaction's latest status; this is the mechanism that the system must implement.
Incorrect Answers
B:For a system whose objective is to personalize repeated interactions without placing all old transcripts in every prompt, explicitly persisted, consent-governed preference data supports bounded long-term personalization; transient chat state should not become an uncontrolled permanent profile; this does not directly meet the required outcome to let an agent retrieve a live order status from the organization's transactional system.
C:For a system whose objective is to resolve a follow-up such as 'What about the second option?' using the user's previous turn, conversation state preserves necessary prior turns for reference resolution; it differs from a searchable repository of long-lived institutional documents; this does not directly meet the required outcome to let an agent retrieve a live order status from the organization's transactional system.
D:For a system whose objective is to support queries mixing specific document codes with natural-language intent, hybrid retrieval runs lexical and vector queries together and merges ranked results; it helps preserve exact tokens while matching semantic concepts; this does not directly meet the required outcome to let an agent retrieve a live order status from the organization's transactional system.
Question 5
A system now includes a component to deploy a multimodal model for visual reasoning. Which evaluation should accompany the rollout?
- Check recall at k for paraphrased questions against known relevant passages
- Score generated functions on passing unit tests and absence of unsafe execution behavior
- Compare retrieval success on mixed identifier-and-paraphrase evaluation queries
- Measure image-grounded answer accuracy against a labeled set of equipment photos
Explanation
Correct Answer
D:The accepted change is to deploy a multimodal model for visual reasoning; this evaluation checks whether it can answer natural-language questions about photographed equipment where the answer depends on visible details, since a multimodal model accepts image evidence alongside text; a text-only model cannot inspect visual input unless a separate vision pipeline supplies it.
Incorrect Answers
A:This test concerns the ability to map queries and reference passages into vectors for semantic retrieval, for which embeddings encode text into vectors used by a similarity search system; they are not themselves a response-generating chatbot; it does not establish whether the change to deploy a multimodal model for visual reasoning works for the current release.
B:This test concerns the ability to produce and review Python functions with useful syntax and unit-test coverage, for which a code-capable generative model can reason over programming context; repository access, tool execution, and tests still need explicit integration; it does not establish whether the change to deploy a multimodal model for visual reasoning works for the current release.
C:This test concerns the ability to support queries mixing specific document codes with natural-language intent, for which hybrid retrieval runs lexical and vector queries together and merges ranked results; it helps preserve exact tokens while matching semantic concepts; it does not establish whether the change to deploy a multimodal model for visual reasoning works for the current release.
Question 6
After deployment, the team observes: A search application only matches exact keywords and misses paraphrases of the stored procedures. Which remediation addresses the likely cause?
- Benchmark a smaller model on labeled tickets and route this bounded task to it
- Run vector retrieval on document chunks using the same embedding space as queries
- Use a model evaluated for coding tasks and validate outputs with an isolated test runner
- Embed the queries and chunks with a compatible model and add vector retrieval
Explanation
Correct Answer
D:The fault signal is that a search application only matches exact keywords and misses paraphrases of the stored procedures; the targeted fix works because embeddings encode text into vectors used by a similarity search system; they are not themselves a response-generating chatbot, restoring the ability to map queries and reference passages into vectors for semantic retrieval.
Incorrect Answers
A:If the failing function were to categorize short support tickets under strict response-time and inference-budget limits, this change could be relevant because a compact model can meet narrow classification needs with fewer inference resources when validated against representative labeled examples; the actual symptom is that a search application only matches exact keywords and misses paraphrases of the stored procedures, which calls for a different repair.
B:If the failing function were to find related guidance when employees use very different wording from the source documents, this change could be relevant because vector search compares embeddings to retrieve semantically close text even when terms differ, provided index and query embeddings are compatible; the actual symptom is that a search application only matches exact keywords and misses paraphrases of the stored procedures, which calls for a different repair.
C:If the failing function were to produce and review Python functions with useful syntax and unit-test coverage, this change could be relevant because a code-capable generative model can reason over programming context; repository access, tool execution, and tests still need explicit integration; the actual symptom is that a search application only matches exact keywords and misses paraphrases of the stored procedures, which calls for a different repair.
Question 7
A system now includes a component to deploy a code-capable model for code-focused assistance. Which evaluation should accompany the rollout?
- Compare classification macro-F1, p95 latency, and per-request inference cost on representative tickets
- Test multi-turn follow-ups for correct reference resolution without cross-user leakage
- Score generated functions on passing unit tests and absence of unsafe execution behavior
- Check recall at k for paraphrased questions against known relevant passages
Explanation
Correct Answer
C:The accepted change is to deploy a code-capable model for code-focused assistance; this evaluation checks whether it can produce and review Python functions with useful syntax and unit-test coverage, since a code-capable generative model can reason over programming context; repository access, tool execution, and tests still need explicit integration.
Incorrect Answers
A:This test concerns the ability to categorize short support tickets under strict response-time and inference-budget limits, for which a compact model can meet narrow classification needs with fewer inference resources when validated against representative labeled examples; it does not establish whether the change to deploy a code-capable model for code-focused assistance works for the current release.
B:This test concerns the ability to resolve a follow-up such as 'What about the second option?' using the user's previous turn, for which conversation state preserves necessary prior turns for reference resolution; it differs from a searchable repository of long-lived institutional documents; it does not establish whether the change to deploy a code-capable model for code-focused assistance works for the current release.
D:This test concerns the ability to map queries and reference passages into vectors for semantic retrieval, for which embeddings encode text into vectors used by a similarity search system; they are not themselves a response-generating chatbot; it does not establish whether the change to deploy a code-capable model for code-focused assistance works for the current release.
Question 8
A release reviewer is assessing a feature intended to use vector similarity search for paraphrases. Which test is most meaningful?
- Assess recall for paraphrases and meaning-preserving rewordings of known questions
- Score generated functions on passing unit tests and absence of unsafe execution behavior
- Track normalized discounted cumulative gain and top-rank relevance on judged query results
- Compare retrieval success on mixed identifier-and-paraphrase evaluation queries
Explanation
Correct Answer
A:The accepted change is to use vector similarity search for paraphrases; this evaluation checks whether it can find related guidance when employees use very different wording from the source documents, since vector search compares embeddings to retrieve semantically close text even when terms differ, provided index and query embeddings are compatible.
Incorrect Answers
B:This test concerns the ability to produce and review Python functions with useful syntax and unit-test coverage, for which a code-capable generative model can reason over programming context; repository access, tool execution, and tests still need explicit integration; it does not establish whether the change to use vector similarity search for paraphrases works for the current release.
C:This test concerns the ability to improve ordering of an already retrieved set by how directly it answers a user's question, for which semantic reranking uses language understanding to reorder candidate search results; it does not replace indexing or generate new source documents; it does not establish whether the change to use vector similarity search for paraphrases works for the current release.
D:This test concerns the ability to support queries mixing specific document codes with natural-language intent, for which hybrid retrieval runs lexical and vector queries together and merges ranked results; it helps preserve exact tokens while matching semantic concepts; it does not establish whether the change to use vector similarity search for paraphrases works for the current release.
Question 9
The acceptance criteria for a customer-facing AI service require it to resolve a follow-up such as 'What about the second option?' using the user's previous turn. Which design should be approved?
- Store approved long-term user preferences separately
- Deploy a code-capable model for code-focused assistance
- Expose an authenticated business API as an agent tool
- Use conversation state to retain dialogue context
Explanation
Correct Answer
D:For the stated requirement to resolve a follow-up such as 'What about the second option?' using the user's previous turn, conversation state preserves necessary prior turns for reference resolution; it differs from a searchable repository of long-lived institutional documents; this is the mechanism that the system must implement.
Incorrect Answers
A:For a system whose objective is to personalize repeated interactions without placing all old transcripts in every prompt, explicitly persisted, consent-governed preference data supports bounded long-term personalization; transient chat state should not become an uncontrolled permanent profile; this does not directly meet the required outcome to resolve a follow-up such as 'What about the second option?' using the user's previous turn.
B:For a system whose objective is to produce and review Python functions with useful syntax and unit-test coverage, a code-capable generative model can reason over programming context; repository access, tool execution, and tests still need explicit integration; this does not directly meet the required outcome to resolve a follow-up such as 'What about the second option?' using the user's previous turn.
C:For a system whose objective is to let an agent retrieve a live order status from the organization's transactional system, an authenticated tool call obtains current application state at request time; stale embeddings cannot guarantee a transaction's latest status; this does not directly meet the required outcome to resolve a follow-up such as 'What about the second option?' using the user's previous turn.
Question 10
An engineer receives this incident report: A support agent treats every follow-up pronoun as a new topic because earlier exchanges are not supplied. What should be changed?
- Persist only allowed preference fields behind identity checks and expiration controls
- Enable semantic ranking over searchable human-readable fields after initial retrieval
- Register an approved order-status API tool and scope its credentials to read-only lookup
- Carry forward the relevant conversation context with the agent's dialogue state
Explanation
Correct Answer
D:The fault signal is that a support agent treats every follow-up pronoun as a new topic because earlier exchanges are not supplied; the targeted fix works because conversation state preserves necessary prior turns for reference resolution; it differs from a searchable repository of long-lived institutional documents, restoring the ability to resolve a follow-up such as 'What about the second option?' using the user's previous turn.
Incorrect Answers
A:If the failing function were to personalize repeated interactions without placing all old transcripts in every prompt, this change could be relevant because explicitly persisted, consent-governed preference data supports bounded long-term personalization; transient chat state should not become an uncontrolled permanent profile; the actual symptom is that a support agent treats every follow-up pronoun as a new topic because earlier exchanges are not supplied, which calls for a different repair.
B:If the failing function were to improve ordering of an already retrieved set by how directly it answers a user's question, this change could be relevant because semantic reranking uses language understanding to reorder candidate search results; it does not replace indexing or generate new source documents; the actual symptom is that a support agent treats every follow-up pronoun as a new topic because earlier exchanges are not supplied, which calls for a different repair.
C:If the failing function were to let an agent retrieve a live order status from the organization's transactional system, this change could be relevant because an authenticated tool call obtains current application state at request time; stale embeddings cannot guarantee a transaction's latest status; the actual symptom is that a support agent treats every follow-up pronoun as a new topic because earlier exchanges are not supplied, which calls for a different repair.