Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 26

Question 250

An experiment is checked every hour and stopped at the first favorable result without a sequential-testing plan. What risk does this introduce?

  1. It removes the need for a control group
  2. The stopping policy can inflate apparent evidence of improvement
  3. It guarantees that the first favorable result is permanent
  4. It proves the candidate has lower latency

Explanation

Correct Answer

B:

Repeatedly checking and stopping selectively changes the interpretation of the result. The experiment needs a valid predefined or sequential analysis approach.

Incorrect Answers

A:

Stopping behavior does not create a comparison baseline. Controlled assignment remains necessary for attributing effects to the candidate.

C:

A temporary favorable fluctuation can disappear with more data. Selective stopping does not establish stable performance.

D:

The stopping rule says nothing about which metric improved or how. Latency needs its own measured comparison and acceptance criterion.

Question 251

A new model improves overall extraction but sharply worsens handwritten invoices, a critical customer segment. How should release evaluation respond?

  1. Approve solely on the global average
  2. Remove handwritten invoices from the reported scope without agreement
  3. Apply the segment's acceptance requirement as well as aggregate metrics
  4. Increase the weight of easy typed invoices

Explanation

Correct Answer

C:

Averaging can hide concentrated regressions. Critical slices need explicit criteria so a broad improvement does not silently sacrifice an important supported workflow.

Incorrect Answers

A:

The average does not satisfy a segment-specific requirement. Users in the harmed category would experience a real regression despite the overall gain.

B:

Changing support commitments requires an explicit product decision. Evaluation should reveal the trade-off rather than hide the affected population.

D:

Reweighting can improve the reported score without improving the failing behavior. The release must reflect the actual workload and commitments.

Question 252

Production documents become longer and use new layouts, while the model and prompts remain unchanged. Which monitoring signal can reveal the shift early?

  1. Changes in input characteristics relative to the validated distribution
  2. Only the model version identifier
  3. Only the total number of successful HTTP responses
  4. Only the average response latency across all document types

Explanation

Correct Answer

A:

A stable implementation can face a changing workload. Tracking relevant input features can identify when old evaluation evidence may no longer represent production.

Incorrect Answers

B:

A fixed model version can encounter inputs unlike its evaluation data. Layout and content changes need monitoring even when no model release occurs.

C:

Transport success does not establish semantic task quality or distribution similarity. Requests can complete normally on unfamiliar inputs.

D:

Latency might change indirectly, but an overall average can conceal the new document distribution. Task-relevant input features provide a more direct signal of this shift.

Question 253

The correctness of a fraud-review recommendation is known only weeks later. What monitoring design best handles this?

  1. Use provisional signals and later reconcile with confirmed outcomes
  2. Replace delayed labels with the model's own confidence
  3. Declare every immediate user approval a true positive
  4. Stop monitoring until every label is available

Explanation

Correct Answer

A:

Early proxies can support operations, but delayed labels are needed to assess real performance. Linking outcomes back to requests enables retrospective validation.

Incorrect Answers

B:

Confidence is not independent outcome evidence. It may be useful only after calibration and cannot substitute for confirmed results.

C:

User approval is not the eventual fraud outcome. Treating it as truth can bias the evaluation toward convenient but incorrect decisions.

D:

Waiting removes useful operational visibility. Provisional metrics can be used if their limits and later reconciliation are explicit.

Question 254

Only highly dissatisfied users submit feedback. The team treats that sample's error rate as the error rate for all traffic. What is the issue?

  1. The feedback population is not representative of all users
  2. Positive responses should be assumed for every silent user
  3. More negative reports automatically remove selection bias
  4. Negative feedback is always factually wrong

Explanation

Correct Answer

A:

Voluntary feedback can concentrate extreme experiences. The team needs representative sampling or an analysis that accounts for who provides feedback.

Incorrect Answers

B:

Silence does not establish satisfaction or correctness. Inferring a positive label for all nonresponders introduces another unsupported assumption.

C:

A larger biased sample can remain unrepresentative. Quantity does not fix a selection process tied to dissatisfaction.

D:

Dissatisfied reports can identify real defects. The limitation is generalizing their frequency to the whole population without a sampling basis.

Question 255

An assistant receives high ratings for persuasive explanations, but expert checks find factual mistakes. Which evaluation principle applies?

  1. The ratings must be discarded in every future analysis
  2. User satisfaction and factual accuracy are separate measures
  3. Whether a rating threshold can replace factual checks for highly rated answers
  4. Whether agreement among satisfied users makes the expert findings irrelevant
  5. High ratings prove that expert checks are unnecessary

Explanation

Correct Answer

B:

Users may appreciate clarity without being able to verify facts. A quality program should measure both experience and evidence-grounded correctness.

Incorrect Answers

A:

Experience data can still be useful. It should be interpreted alongside correctness measures rather than promoted to a complete quality verdict.

C:

A satisfaction threshold still measures the same user response that failed to reveal errors. It cannot substitute for factual validation.

D:

Shared satisfaction can coexist with the same overlooked error. Expert evidence of incorrect facts must be evaluated on its merits.

E:

Ratings do not establish the truth of claims. The observed disagreement shows why independent factual evaluation remains important.

Question 256

A system sends thousands of low-severity alerts, and operators miss the few incidents requiring intervention. What should be improved?

  1. Increase every alert's urgency to critical
  2. Disable all monitoring to reduce interruptions
  3. Send identical alerts to more unassigned channels
  4. Prioritize actionable signals with ownership and escalation criteria

Explanation

Correct Answer

D:

An alert should connect a meaningful condition to an appropriate response. Excess noise can overwhelm attention even when the system technically reports every event.

Incorrect Answers

A:

Uniform urgency destroys prioritization. It does not help operators distinguish the few events that need immediate action.

B:

Removing visibility hides consequential incidents too. The goal is a useful signal-to-noise ratio, not absence of observation.

C:

Broadcasting does not establish responsibility. It can multiply noise while leaving no person or team accountable for acting.

Question 257

Two extractors make the same number of mistakes. One misreads decorative headings; the other changes payment amounts. What should their evaluation reflect?

  1. The combined error count without distinguishing consequences
  2. Equal error counts prove equal production risk
  3. The differing consequences of error types
  4. Hide low-severity mistakes to make reporting simpler

Explanation

Correct Answer

C:

Equal counts can hide unequal business harm. A task-appropriate rubric should distinguish severity while still reporting the underlying error categories transparently.

Incorrect Answers

A:

Two systems with equal error counts can cause very different harm. A business-weighted comparison must preserve the severity of the relevant failure types.

B:

This ignores the effect of each mistake. A harmless formatting defect and a wrong payment amount can have very different consequences.

D:

Simplification should not erase evidence. Reporting categories and consequences provides a clearer picture than selectively omitting inconvenient detail.

Question 258

Model token cost falls after a change, but total operating cost rises because tool calls and human reviews increase. What cost metric is needed?

  1. Only the cheapest stage in each configuration
  2. Only the model provider's token invoice
  3. Only the number of generated responses
  4. End-to-end cost including dependencies and review work

Explanation

Correct Answer

D:

Local savings can be offset elsewhere in the workflow. A production cost comparison should include the resources required to deliver a usable outcome.

Incorrect Answers

A:

Selecting favorable components hides the complete workflow. The decision needs a consistent boundary for all relevant costs.

B:

That captures one component while omitting the costs causing the increase. It cannot explain the total operating expense.

C:

Response count does not reflect variable tool usage or human effort. Equal volumes can have very different unit economics.

Question 259

A batch configuration completes more documents per hour but makes each interactive user wait longer. What does this demonstrate?

  1. Throughput and per-request latency are distinct objectives
  2. Only total daily volume is relevant to interactive users
  3. Higher throughput guarantees lower latency for every user
  4. The latency measurement must be invalid

Explanation

Correct Answer

A:

Higher aggregate completion rate can coexist with longer individual waits. The appropriate configuration depends on the workload's service requirements.

Incorrect Answers

B:

Interactive experience depends on response time as well as volume. A service can process many requests and still feel slow to each user.

C:

The measurements directly contradict that assumption. Batching and queueing can improve resource efficiency while increasing waiting time.

D:

Different metrics can move in opposite directions legitimately. The team should inspect the design trade-off before rejecting either observation.

Question 260

A dashboard excludes every timed-out request from its success-rate calculation. Why is the resulting availability figure misleading?

  1. Including timeouts would count the same request twice
  2. Only requests with long answers should count
  3. It removes a class of failed user experiences from the denominator
  4. Timeouts are always successful because no error body exists

Explanation

Correct Answer

C:

Timeouts are part of the service outcome. Excluding them can inflate apparent success and conceal the failures the metric should reveal.

Incorrect Answers

A:

A well-defined request outcome can count each request once. Proper classification does not require duplication.

B:

Answer length is unrelated to whether the service was available. The denominator should follow the agreed population of eligible requests.

D:

Absence of an error body does not establish completion. The user may have received no usable result within the required time.

Claude Certified Architect - Professional (CCAR-P) Practice Tests