Question 20
A platform exposes 600 internal tools, but each support task uses only a few. Loading every detailed schema consumes most input tokens and confuses selection. Which design should be evaluated?
- Cache the complete tool catalog but still expose every schema for each decision
- Raise the context limit while retaining all detailed schemas in every request
- Use task-directed tool discovery and load the selected detailed interfaces only when needed
- Rotate through fixed catalog batches without matching them to the current task
Explanation
Correct Answer
C:A discovery step narrows the active interface set before execution. It reduces irrelevant input while preserving access to capabilities when the task actually requires them.
Incorrect Answers
A:Caching can reduce repeated input processing costs, but does not narrow the set of interfaces the model must distinguish. The tool-selection confusion remains.
B:A larger context may fit the catalog without addressing its relevance or the ambiguity between similar tools. Capacity alone does not solve capability bloat.
D:Fixed batches can exclude the needed interface or require several additional searches. Task-directed discovery is better matched to selecting a small relevant subset.
Question 21
A document assistant scores well on examples copied from its prompt demonstrations but fails on new document layouts. What evaluation change most directly tests generalization?
- Measure extraction accuracy after randomly splitting repeated prompt examples into two sets
- Allow the production model to generate both evaluation inputs and reference labels
- Add reordered copies of familiar documents to increase the evaluation sample size
- Evaluate unseen representative document layouts and edge cases against independently checked reference labels
Explanation
Correct Answer
D:Unseen representative cases test whether the approach transfers beyond the demonstrations. Edge cases should reflect meaningful production risks rather than arbitrary difficulty.
Incorrect Answers
A:Random splitting cannot make duplicated examples independent. Familiar content can leak across the split and overstate performance on new layouts.
B:Using one model to invent inputs and their truth labels can reproduce its own misconceptions. Grounded reference labels and representative unseen documents are needed for a meaningful test.
C:Reordering known documents changes presentation order, not the range of layouts tested. A larger count does not fix an unrepresentative evaluation set.
Question 22
A generated case summary must preserve exact account identifiers and accurately explain the customer's complaint. Which test combination best covers both requirements?
- Use deterministic identifier checks and a readability-only rubric for the complaint
- Validate the JSON schema and compare summary length with the reference summary
- Grade meaning by embedding similarity without separately checking account identifiers
- Check identifiers exactly and assess complaint fidelity against the source
Explanation
Correct Answer
D:Exact fields can be checked programmatically, while meaning and coverage need a content-oriented assessment tied to the source. Human calibration helps establish whether semantic grades reflect the intended standard.
Incorrect Answers
A:This combination verifies identifiers but scores how the complaint reads rather than whether it is accurate. A fluent factual error can still pass the readability rubric.
B:Schema and length checks constrain structure but not the correctness of the values or narrative. Two equally long summaries can describe different complaints.
C:Semantic similarity can remain high when a single critical identifier changes. Exact fields require a check sensitive to that precise difference.
Question 23
A team wants to compare two prompt versions for a live support assistant. Customer complexity varies by day. Which experiment most credibly attributes any improvement to the prompt?
- Let support agents choose the version for each case and compare the observed resolution rates
- Change retrieval settings together with the prompt and attribute the combined result to prompting
- Randomize eligible cases across recorded prompt variants using predefined quality and latency measures
- Run A during a normal week and B during a holiday surge, comparing their mean scores
Explanation
Correct Answer
C:Random assignment helps distribute workload differences across variants. Predefined measures and configuration records make the comparison interpretable and reduce selective reporting.
Incorrect Answers
A:Self-selection can place easier or preferred cases into one group. Raw totals would mix prompt effects with user and workload differences.
B:A simultaneous retriever change creates another plausible cause for the outcome. Keeping other relevant configuration fixed makes the prompt comparison interpretable.
D:Day-specific demand can confound the comparison. A holiday surge may change case difficulty and staffing independently of the prompt.
Question 24
A retrieval test shows the correct policy passage is present in the model input, but answers repeatedly ignore its explicit exception. Other passage types work well. What is the best next diagnostic step?
- Test prompt changes on exception cases with the model and retrieval held fixed
- Increase the retrieved passage count and switch the model in the same experiment
- Rebuild the index with a different embedding model before testing instruction changes
- Route every case to a larger model without measuring the exception error separately
Explanation
Correct Answer
A:The available evidence narrows the problem to how supplied information is used. A controlled exception-focused evaluation can test whether the prompt makes the decision rule sufficiently clear.
Incorrect Answers
B:Changing context volume and model choice together obscures which change helps. The correct passage is already available, so a narrower test better isolates its interpretation.
C:Index changes target missing or poorly ranked evidence. The observed failure is repeated disregard of evidence already present, so this is a less direct initial diagnostic.
D:A larger model may improve some tasks, but the change needs evidence on the identified error type. Broad rerouting without an exception measure can hide the unresolved cause.
Question 25
An assistant meets a 2-second average response time, but the slowest 5% of requests take more than 15 seconds and trigger user abandonment. Which monitoring change best reflects the problem?
- Track end-to-end upper percentiles and the slow request paths
- Report hourly mean latency to increase the resolution of the existing average
- Track median latency and use it as the primary abandonment indicator
- Measure model-only percentiles while excluding queue and tool time
Explanation
Correct Answer
A:Percentile measures expose the slow fraction hidden by the average. Segmenting traces by tool use, retries, or input size helps locate the paths responsible.
Incorrect Answers
B:Shorter averaging intervals may reveal broad peaks, but averages can still hide a slow minority within every hour. The observed problem requires a measure of the distribution’s upper tail.
C:The median describes the middle request, while the complaint concerns the slowest fraction. It may remain stable even when those users experience substantial delay.
D:Model percentiles are useful for diagnosing one component but omit other time the user experiences. The service measure must include the full request path.
Question 26
Variant A costs $0.02 per attempt and completes 50% of cases correctly. Variant B costs $0.03 per attempt and completes 90% correctly. Ignoring all other costs, which conclusion follows from cost per correct completion?
- A: $0.010; B: $0.027 per correct completion, so A costs less
- A: $0.020; B: $0.030 per correct completion, so A costs less
- A: $0.040; B: $0.300 per correct completion, so A costs less
- A: $0.040; B: about $0.033 per correct completion, so B costs less
Explanation
Correct Answer
D:Dividing attempt cost by success probability gives 0.03/0.90 and 0.02/0.50. This narrow comparison favors B; a full production decision would also include review, retry, latency, and failure costs.
Incorrect Answers
A:These figures multiply call cost by success probability. Cost per useful outcome instead divides total spending by the number of successful outputs.
B:The figures are per-attempt prices and omit unsuccessful attempts. Across many cases, those attempts consume money without increasing the number of correct completions.
C:The value for A uses its success fraction, while the value for B uses its failure fraction. Both variants must be evaluated with the same denominator: correct completions.
Question 27
A retrieved vendor page contains text instructing the assistant to upload internal purchase records to an external URL. The page is relevant evidence for product specifications. Which response design best handles this?
- Ask the model to classify the page as safe, then permit its requested upload if it passes
- Strip known injection phrases and treat the remaining page text as trusted instructions
- Block this particular URL but allow tool calls requested by other retrieved pages
- Treat all retrieved content as untrusted and enforce external-action permissions in application code
Explanation
Correct Answer
D:The product facts can still be considered as evidence, while embedded action requests remain untrusted. External side effects need separate controls so a retrieved page cannot grant itself access to internal records.
Incorrect Answers
A:A model safety classification is a fallible signal, not authorization to release internal records. Granting action authority after that classification leaves the trust boundary exposed.
B:Pattern filtering can miss paraphrases or novel attacks. The remaining text still comes from an external source and cannot grant permission to upload internal data.
C:Blocking a known malicious source can help containment, but the same attack can arrive from another page. The architecture must control the authority of all retrieved content.
Question 28
A reviewer approves a draft customer notification, but the assistant can edit the message afterward before sending it. The policy requires approval of the exact sent content. Which control closes the gap?
- Bind execution approval to the immutable reviewed content and require reapproval after changes
- Store reviewer identity and approval time but allow the message body to change
- Protect the approved message identifier while allowing its stored content to be overwritten
- Log post-approval edits and send the latest version under the original approval
Explanation
Correct Answer
A:The executor can verify that the approved artifact and outgoing artifact are the same version. A changed body invalidates the earlier approval instead of silently extending it to new content.
Incorrect Answers
B:Identity and time establish who approved something and when, but not whether the outgoing body is the reviewed body. The authorization must be tied to a content version.
C:An unchanged identifier is insufficient if the object it names is mutable. Versioning or a content-integrity check must detect a replacement body.
D:Logging supports later investigation but does not enforce prior approval of the sent content. A changed message needs a new approval before transmission.
Question 29
A support summarizer needs the issue description and product type. The source record also contains payment card data that is irrelevant to summarization. What is the best input design?
- Filter the irrelevant sensitive fields before constructing and transmitting the model request
- Encrypt the full record in transit and send every field to the model
- Send the record, then redact sensitive fields from the generated response
- Retain every field in the request but shorten the downstream log-retention period
Explanation
Correct Answer
A:Filtering at the application boundary prevents irrelevant sensitive fields from entering this processing path. The remaining issue description and product type provide the information the summarizer actually requires.
Incorrect Answers
B:Transport encryption protects data on the network but does not minimize what is processed at the destination. The unnecessary fields still enter the model request.
C:Output redaction can reduce disclosure in the answer, but it occurs after the model has received the data. The stated task does not need those fields in the first place.
D:Shorter retention can reduce how long diagnostic copies remain, but does not remove unnecessary data from the request. Input minimization addresses a different stage.
Question 30
A safety classifier blocks 98% of a labeled attack set, but a new threshold also blocks 20% of legitimate account-recovery requests. The team must retain useful service while controlling attack risk. What evidence should guide the next mitigation?
- Keep the threshold because aggregate attack detection is the only release measure
- Lower the threshold until legitimate refusals disappear, then release without attack retesting
- Choose a cutoff from the score distribution without labeling blocked requests
- Compare attack misses and legitimate blocks by slice before selecting a mitigation
Explanation
Correct Answer
D:A labeled, sliced evaluation makes the trade-off between false positives and missed attacks visible. The team can then test threshold, classifier, or workflow changes against explicit security and usability criteria.
Incorrect Answers
A:Attack detection is important, but unintended blocking also affects whether the service works for its intended users. The release decision needs evidence on both error types.
B:Reducing false positives may increase missed attacks. A candidate threshold must be evaluated on malicious cases as well as the legitimate traffic it unblocks.
C:A score histogram can show how often the classifier fires, but cannot distinguish harmful requests from legitimate ones. Selecting a cutoff requires outcome labels and an explicit trade-off between the two error types.