Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 25

Question 240

Throughput stops increasing as concurrency rises, while queue delay grows sharply. What is the likely interpretation?

  1. A bottleneck has reached capacity and extra concurrency adds waiting
  2. The system needs longer queues as its only fix
  3. The model's answers are necessarily becoming less grammatical
  4. Every additional request is improving processing efficiency

Explanation

Correct Answer

A:

Once a limiting resource saturates, more arrivals can increase latency without increasing completed work. Measurement should identify the constrained stage.

Incorrect Answers

B:

A larger queue can absorb a temporary burst but cannot increase sustained service capacity. It may simply permit longer waiting times.

C:

The observed metrics concern capacity, not language quality. Content changes would require separate evidence rather than inference from queue growth.

D:

The flat throughput contradicts that claim. More concurrent work is accumulating as delay rather than being completed faster.

Question 241

A benchmark measures only warmed connections and populated caches, but first requests after deployment are much slower. What should be reported?

  1. Only the cold result for every future request
  2. A blended value without identifying the traffic proportions
  3. Separate cold-start and steady-state performance
  4. Only the warmed result because it is faster

Explanation

Correct Answer

C:

Both conditions affect users in different operating periods. Reporting them separately avoids hiding initialization costs behind a warmed average.

Incorrect Answers

A:

Cold performance is also not representative of all traffic. The report should show the relevant conditions rather than substitute one for the other.

B:

An unexplained blend is difficult to interpret or reproduce. The measurement needs a defined mix or separate condition-specific results.

D:

Selecting the favorable condition misrepresents the deployment experience. First-request latency can matter to availability and user expectations.

Question 242

All configurations meet a 93% quality minimum on the same workload. A scores 96%, takes 1.2 seconds, and costs 0.06 per task; B scores 96%, takes 1.8 seconds, and costs 0.08; C scores 98%, takes 2 seconds, and costs 0.12; D scores 94%, takes 1 second, and costs 0.04. Higher quality, lower latency, and lower cost are preferred. Which conclusion follows?

  1. A is dominated by B
  2. D dominates every other configuration
  3. B is dominated by A
  4. C is dominated by A
  5. D is dominated by A

Explanation

Correct Answer

C:

A matches B on quality while improving both latency and cost. On the stated measurements, B offers no compensating advantage over A.

Incorrect Answers

A:

B has the same quality but worse latency and cost. The direction of dominance is reversed; B does not improve any stated objective over A.

B:

D leads on latency and cost but has the lowest quality score. Passing a minimum does not make all above-threshold quality values identical under the stated preferences.

D:

A is faster and cheaper, but C has higher quality. That remaining quality advantage prevents A from dominating C under the three stated objectives.

E:

A has higher quality, but D is faster and cheaper. Both satisfy the minimum, so the remaining differences represent a trade-off rather than dominance.

Question 243

A safety evaluation tests only obvious forbidden phrases. Which addition improves its ability to reveal realistic weaknesses?

  1. A guarantee based on the number of test cases
  2. Only benign requests with polite wording
  3. Paraphrases, contextual attacks, and multi-step attempts under controlled testing
  4. More copies of the same obvious phrases

Explanation

Correct Answer

C:

Attack wording and context can vary beyond known strings. Diverse tests probe the underlying behavior rather than only recognition of a fixed vocabulary.

Incorrect Answers

A:

A large finite set does not prove universal safety. Coverage should be reasoned about in terms of threats and failure modes.

B:

Benign cases are needed for false-positive measurement but cannot replace adversarial coverage. Both sides of the boundary require testing.

D:

Duplication increases count without expanding the attack space. It can inflate confidence while leaving the same untested weaknesses.

Question 244

A controlled adversarial test reveals a data-exfiltration path, and a fix is implemented. What should happen to that scenario afterward?

  1. Retain a safe reproducible regression case
  2. Retain a regression that contacts the original destination using production secrets
  3. Delete the scenario because the incident is resolved
  4. Replace it with a generic successful conversation

Explanation

Correct Answer

A:

A durable test can detect reintroduction of the same failure during later changes. It should preserve the relevant attack mechanism without exposing real secrets.

Incorrect Answers

B:

A regression should reproduce the failure mechanism under controlled conditions. Real credentials and an uncontrolled destination can turn verification into another exposure.

C:

Removal loses evidence that the fix continues to work. Future changes could recreate the weakness without a targeted signal.

D:

A happy-path case does not exercise the discovered failure. The regression should preserve the condition that made the original vulnerability visible.

Question 245

Developers repeatedly tune prompts on the same supposed final test set. What should they do before making an unbiased final performance claim?

  1. Increase the score precision to more decimal places
  2. Report only the last improved score
  3. Rename the existing set final without changing usage
  4. Evaluate on a fresh held-out set not used for tuning

Explanation

Correct Answer

D:

Once scores guide development, the set functions as development data. A separate untouched evaluation is needed for a less biased final estimate.

Incorrect Answers

A:

Numerical precision does not remove selection bias. The estimate can be precisely measured on an overused test set and still be misleading.

B:

The final score reflects the tuning history. Omitting that history does not make it an independent estimate of generalization.

C:

A label does not restore independence. The team's repeated optimization has already influenced performance on those cases.

Question 246

Users receive several related assistant interactions, and exposure to one variant changes later behavior. How should an A/B test avoid mixing treatments within a user?

  1. Assign all early users to A and all later users to B
  2. Assign by whichever response finishes first
  3. Randomize at the user or appropriate cluster level
  4. Let users switch variants after seeing the answer

Explanation

Correct Answer

C:

Related interactions are not independent treatment units. Consistent assignment reduces cross-variant contamination while analysis accounts for the chosen clustering.

Incorrect Answers

A:

Chronological assignment can confound the variant with changes over time. The experiment should randomize at a unit that respects repeated-user effects.

B:

Completion timing is affected by the system and workload. It is not a controlled random allocation of comparable users.

D:

Self-selection can correlate treatment with satisfaction or difficulty. It makes the observed groups less comparable for causal inference.

Question 247

A large experiment finds a statistically detectable quality gain of 0.01%, accompanied by a 40% cost increase. What additional judgment is needed?

  1. Any statistically detectable change must be deployed
  2. Whether the trial can exclude high-cost users after results are known
  3. Whether a larger sample can turn the same 0.01% gain into a larger business effect
  4. Cost is irrelevant once a p-value is small
  5. Whether the effect is practically valuable under the business constraints

Explanation

Correct Answer

E:

Statistical detectability does not establish worthwhile benefit. The decision must consider effect size, costs, risks, and the use case's meaningful improvement threshold.

Incorrect Answers

A:

A tiny gain can be uneconomic or irrelevant. The test result does not override the service's cost and operational requirements.

B:

Post hoc exclusions can distort the decision population and its economics. The small gain must be assessed against relevant costs for the intended users.

C:

A larger sample can improve precision without increasing the underlying effect. It does not make the observed gain more valuable relative to the stated cost increase.

D:

Statistical evidence concerns the observed difference, not the desirability of its price. Business trade-offs remain part of the release decision.

Question 248

A prespecified paired comparison estimates that candidate A improves accuracy over B by 0.3 percentage points. A valid 95% confidence interval for the paired difference is -1.2 to +1.8 percentage points. What conclusion is supported at that confidence level?

  1. The slightly higher point estimate proves the better model
  2. The comparison does not establish a directional accuracy advantage
  3. The candidates are proven identical in every respect
  4. The upper endpoint establishes that A improves accuracy by 1.8 points

Explanation

Correct Answer

B:

The interval for the difference includes zero and values favoring either candidate. It does not establish superiority at the stated confidence level, and it does not prove equivalence.

Incorrect Answers

A:

A point estimate alone ignores uncertainty. A small numerical lead can reverse when additional representative cases are evaluated.

C:

Failure to establish a difference is not proof of equivalence. An equivalence claim requires an appropriate design and acceptable difference bounds.

D:

An interval endpoint is not the estimated effect. Selecting its most favorable value ignores the rest of the interval, including zero and negative differences.

Question 249

A team tries 100 prompt variants on one dataset and reports only the best improvement. What should the reviewer examine?

  1. The winner's score on the same set used to select it
  2. The number of candidates, treated as unrelated to selection uncertainty
  3. Selection effects and validation on independent data
  4. Whether the winning variant also has the best rank on a reshuffle of the same cases

Explanation

Correct Answer

C:

Searching many candidates can produce an apparent winner through chance or overfitting. Independent confirmation helps test whether the selected gain persists.

Incorrect Answers

A:

The selection process has already favored the strongest observed score on that set. Independent validation is needed to assess whether the advantage generalizes.

B:

Testing more candidates increases the opportunity to select a favorable random result. The evaluation should account for that selection effect.

D:

Reordering the selection data does not create independent evidence. It retains the same cases on which the best variant was chosen.

Question 250

An experiment is checked every hour and stopped at the first favorable result without a sequential-testing plan. What risk does this introduce?

  1. It removes the need for a control group
  2. The stopping policy can inflate apparent evidence of improvement
  3. It guarantees that the first favorable result is permanent
  4. It proves the candidate has lower latency

Explanation

Correct Answer

B:

Repeatedly checking and stopping selectively changes the interpretation of the result. The experiment needs a valid predefined or sequential analysis approach.

Incorrect Answers

A:

Stopping behavior does not create a comparison baseline. Controlled assignment remains necessary for attributing effects to the candidate.

C:

A temporary favorable fluctuation can disappear with more data. Selective stopping does not establish stable performance.

D:

The stopping rule says nothing about which metric improved or how. Latency needs its own measured comparison and acceptance criterion.

Claude Certified Architect - Professional (CCAR-P) Practice Tests