Anthropic CCA-F (Claude Certified Architect - Foundations (CCAR-F)) Practice Test 35

Question 340

An LLM grader marks verbose but incorrect answers as excellent. What should the team do before relying on its scores?

  1. Remove all incorrect examples from the grader's evaluation set.
  2. Increase the maximum answer length so every answer looks excellent.
  3. Use the grader's confidence as its only validation.
  4. Calibrate the rubric against expert-labeled examples, including fluent wrong answers and concise correct ones.

Explanation

Correct Answer

D:

Contrastive calibration tests whether the grader rewards the intended criteria rather than superficial verbosity. Human-calibrated counterexamples should demonstrate that factual mistakes outweigh polished presentation when that is the rubric. Grader agreement can then be checked before using its scores for release decisions.

Incorrect Answers

A:

Excluding failures prevents measuring whether the grader can recognize them.

B:

Longer answers reinforce the grader's unwanted preference without improving factual quality.

C:

Self-reported confidence does not demonstrate agreement with the intended quality standard.

Question 341

A report has both a required JSON field set and a subjective clarity criterion. Which evaluation design is appropriate?

  1. Use only a subjective impression to decide whether every required key exists.
  2. Use one overall reviewer score without separately checking the required field set.
  3. Use only a JSON parser to decide whether prose is clear for its audience.
  4. Use deterministic checks for structure and a calibrated rubric for clarity.

Explanation

Correct Answer

D:

Different criteria need assessment methods that can actually observe the property being evaluated. Exact structural requirements lend themselves to parsers and assertions. Audience clarity needs a rubric grounded in reader needs, with examples that make its scoring consistent.

Incorrect Answers

A:

A deterministic field check is more direct and reproducible for an exact structural requirement.

B:

An aggregate impression can overlook a missing machine-required field. Structural requirements deserve direct checks independent of prose quality.

C:

Parsing cannot measure whether a human reader can understand the explanation.

Question 342

Prompt A passes 19 of 20 examples and Prompt B passes 18. What is the most defensible interpretation?

  1. A is guaranteed to outperform B on every future workload.
  2. The two prompts are mathematically identical because their scores are close.
  3. B is unusable because it missed two examples.
  4. A scored higher here; superiority needs more representative evidence.

Explanation

Correct Answer

D:

A one-example difference in a small sample may not reflect a stable population-level advantage. The evaluator should also inspect which cases differ and how representative the sample is. A small numerical edge can hide a serious regression on an important scenario.

Incorrect Answers

A:

The observed sample does not justify a universal claim about unseen cases.

B:

Similar aggregate scores do not imply identical behavior or error patterns.

C:

Usability depends on error types, requirements, and broader evidence rather than this count alone.

Question 343

Two classifiers have equal overall accuracy. One misses rare critical alerts; the other sends more harmless cases to review. What should the decision rubric include?

  1. Explicit costs and acceptance limits for the distinct false-negative and false-positive outcomes.
  2. The model name's popularity as a substitute for error analysis.
  3. Only the total number of predictions, because the error rates are equal.
  4. A rule that every error has identical cost regardless of the application.

Explanation

Correct Answer

A:

The product's error consequences determine whether the two equally accurate systems are operationally equivalent. A rare critical miss can carry more consequence than several unnecessary reviews. Separate error budgets make that tradeoff visible instead of hiding it inside one accuracy percentage.

Incorrect Answers

B:

Popularity does not quantify the consequences of the observed mistakes.

C:

Equal aggregate accuracy does not imply equal consequences for different error types.

D:

The scenario explicitly distinguishes critical misses from additional harmless review.

Question 344

A document assistant passes normal examples but follows instructions embedded in malicious source text. Which evaluation addition directly addresses the gap?

  1. More clean documents containing the same benign wording.
  2. Cases with hostile source instructions and expected task-preserving behavior.
  3. Only longer final answers to the existing clean examples.
  4. Removal of all source material from every production request.

Explanation

Correct Answer

B:

Adversarial source examples test whether the assistant maintains the intended data/instruction boundary. The tests should include documents that remain useful evidence while containing malicious instructions. Passing requires answering the legitimate question without promoting those instructions into authority.

Incorrect Answers

A:

Additional ordinary cases do not exercise the observed instruction-confusion failure mode.

C:

Output length does not test resistance to malicious document instructions.

D:

Eliminating documents defeats the assistant's purpose instead of evaluating safe document use.

Question 345

A policy changes the correct category for a class of tickets. The evaluation still uses labels from the old policy. What should happen before interpreting the new prompt's score?

  1. Delete the entire evaluation history so the old policy is forgotten.
  2. Ask the model to choose whichever label produces a higher score.
  3. Treat agreement with obsolete labels as the sole measure of current correctness.
  4. Version and update affected ground-truth labels, then compare systems against the intended policy version.

Explanation

Correct Answer

D:

A score is meaningful only relative to the decision rule its reference labels encode. Old results remain meaningful for the old policy when their version is retained. Current release judgments need labels that describe the intended current behavior.

Incorrect Answers

A:

Previous results remain interpretable when their policy and label versions are retained.

B:

Optimizing against known obsolete labels does not satisfy the new policy.

C:

Historical labels no longer represent the changed decision rule for affected cases.

Question 346

A classifier reaches 99% accuracy by answering only the easiest 10% of requests and abstaining on the rest. Which TWO metrics should accompany accuracy? Choose TWO.

  1. The model's preferred output style.
  2. The downstream cost or disposition of abstained cases.
  3. Coverage or answer rate across the full workload.
  4. Only the accuracy of the single easiest example.
  5. Only the response latency on the answered easiest cases.

Explanation

Correct Answers

B:

Abstentions may require human work, delays, or failures that affect the overall product outcome.

C:

Coverage reveals how often the system actually makes a decision rather than abstaining.

Incorrect Answers

A:

Style preference is unrelated to the operational effect of abstention.

D:

One selected example cannot characterize the system's useful workload coverage.

E:

Latency on accepted cases does not show how much of the total workload was left unresolved.

Question 347

A fraud-review classifier misses too many true review cases, while the team can tolerate more false alarms. Which evaluation change is most relevant?

  1. Report only the number of correctly ignored ordinary cases.
  2. Optimize only overall accuracy on a dataset dominated by non-fraud cases.
  3. Increase the required score threshold without measuring recall.
  4. Evaluate threshold choices using recall, precision, and the team's stated error costs.

Explanation

Correct Answer

D:

A threshold should reflect the tradeoff between missed positives and additional review workload. Lowering a decision threshold may improve recall at the cost of more false positives, but the actual tradeoff should be measured. The selected operating point must fit the team's review capacity and missed-case tolerance.

Incorrect Answers

A:

That count excludes the missed review cases driving the requirement.

B:

Majority-class accuracy can hide missed positive cases and the cost of those misses.

C:

A stricter threshold commonly reduces flagged cases and may worsen the stated problem.

Question 348

An application confuses a model's context window with its maximum generated-output limit. Which distinction should guide request sizing?

  1. The applicable context budget and output limit must both be respected for the selected model.
  2. A large context window guarantees an equally large single response.
  3. Context capacity and output capacity can be inferred from the same character count.
  4. The output limit can always be bypassed by asking for a longer answer.

Explanation

Correct Answer

A:

A request can fit its input budget yet still need an output allowance within the model's separate generation constraint. The precise accounting rules depend on the selected model and features, so the client should use their documented limits. Raising a requested response length cannot override a service-enforced maximum.

Incorrect Answers

B:

Input capacity and output limits are different constraints and should not be assumed identical.

C:

Character length is not a reliable shared proxy for the selected model's token-based constraints.

D:

A natural-language request does not override service-enforced generation limits.

Question 349

A prompt fills nearly all available context with retrieved documents, leaving too little capacity for the required answer. What should the application change?

  1. Delete the user's question while preserving every retrieved passage.
  2. Allocate space for the expected response before selecting how much evidence to include.
  3. Assume the model will compress all input automatically without consequences.
  4. Increase the number of retrieved documents further.

Explanation

Correct Answer

B:

Budgeting input and output together prevents retrieval from consuming the capacity needed to complete the task. Evidence selection should retain the passages needed for the answer while leaving generation room. Filling the input allocation is not a success if the response cannot finish within the remaining budget.

Incorrect Answers

A:

The question is essential task context and should not be sacrificed to retain irrelevant material.

C:

Implicit compression is not a reliable substitute for deliberate request budgeting.

D:

More input consumes additional budget and worsens the shortage available for the answer.

Question 350

A developer estimates token use from the user message alone, but requests still exceed the budget. Which TWO components should the preflight count also consider? Choose TWO.

  1. Only cached-token billing discounts, subtracting those tokens from logical context use.
  2. System instructions and tool definitions included in the request.
  3. Only the final answer's character count from an earlier unrelated task.
  4. Conversation history and tool results actually sent with the request.
  5. Only the filename of each input document instead of its included content.

Explanation

Correct Answers

B:

These prompt components contribute tokens even though they are not part of the visible user message.

D:

Prior messages and observations can occupy substantial context beyond the latest user text.

Incorrect Answers

A:

A billing discount does not mean the cached prompt content ceases to occupy the model's request context.

C:

An unrelated response length does not measure this request's input footprint.

E:

A filename does not represent the tokens consumed by document text sent to the model.

Claude Certified Architect - Foundations (CCAR-F) (CCA-F) Practice Tests