Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 23

Question 220

A proposed model change cuts costs but may affect only complex customer inquiries. What should the solution architect prioritize?

  1. Test the components separately or use a planned factorial experiment.
  2. Set sample size, endpoints and stopping rules before the test.
  3. Stratify A/B results by request difficulty and inspect critical cohorts.
  4. Deploy by canary stages with monitoring and rollback triggers.

Explanation

Correct Answer

C:

Average savings may hide degraded service for the tasks that most need advanced reasoning. Predeclare segment-aware model experiment as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

This shifts attention toward experimental isolation of interventions when the relevant decision is about segment-aware model experiment. Unless the architecture can stratify A/B results by request difficulty and inspect critical cohorts, the failure described in the stem remains unresolved.

B:

For a workflow whose primary weakness was precommitted experiment stopping, that proposal could be reasonable. Here the unmet obligation is segment-aware model experiment, and that requires the system to stratify A/B results by request difficulty and inspect critical cohorts.

D:

It is aimed at a different outcome, safe staged experimentation. No evidence is given that it provides segment-aware model experiment; the needed mechanism is to stratify A/B results by request difficulty and inspect critical cohorts at the appropriate boundary.

Question 221

A pilot changes both retrieval ranking and output format, then attributes gains to the ranking change. Which design choice best matches the stated constraint?

  1. Use safety guardrails as stopping criteria alongside the primary business metric.
  2. Use matched cohorts or random assignment and account for calendar effects.
  3. Control multiple-comparison risk and verify the selected candidate on fresh data.
  4. Test the components separately or use a planned factorial experiment.

Explanation

Correct Answer

D:

Changing multiple factors without design prevents clean attribution to either modification. Predeclare experimental isolation of interventions as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

There is a legitimate use case for constrained optimization in experiments, but it does not supply the test or enforcement needed for experimental isolation of interventions. The design needs to test the components separately or use a planned factorial experiment.

B:

Consider its actual operating scope: control for observational confounding. The requested assurance is experimental isolation of interventions, which calls for an implementation that can test the components separately or use a planned factorial experiment, not merely for a related improvement.

C:

The alternative could reduce risk associated with reliable multi-variant evaluation, yet the described behavior requires experimental isolation of interventions. That outcome depends on the capability to test the components separately or use a planned factorial experiment.

Question 222

An experimental prompt increases conversion but also increases unsupported product claims. Which architecture adjustment is most appropriate?

  1. Randomize eligible traffic and compare predefined business and safety metrics.
  2. Use safety guardrails as stopping criteria alongside the primary business metric.
  3. Test the components separately or use a planned factorial experiment.
  4. Set sample size, endpoints and stopping rules before the test.

Explanation

Correct Answer

B:

Performance gains do not justify crossing nonnegotiable compliance thresholds. Predeclare constrained optimization in experiments as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

By itself, this improves causal evidence of prompt improvement, but the case is not asking the architect to optimize that dimension. The requirement is constrained optimization in experiments, supported by an approach to use safety guardrails as stopping criteria alongside the primary business metric.

C:

The option might improve experimental isolation of interventions; that remains insufficient for constrained optimization in experiments. The case is resolved by constrained optimization in experiments, which depends on a design that will use safety guardrails as stopping criteria alongside the primary business metric.

D:

The proposed action prioritizes precommitted experiment stopping. That can coexist with the right solution, but it cannot replace constrained optimization in experiments: the application must still use safety guardrails as stopping criteria alongside the primary business metric.

Question 223

A low-traffic legal workflow cannot produce statistically useful A/B evidence in one week. Which next step is most defensible?

  1. Use safety guardrails as stopping criteria alongside the primary business metric.
  2. Randomize eligible traffic and compare predefined business and safety metrics.
  3. Use offline paired evaluation and longer controlled observation before making strong claims.
  4. Include effect size and lifecycle expense in the adoption decision.

Explanation

Correct Answer

C:

Sparse data requires a study design that acknowledges uncertainty rather than premature win declarations. Predeclare appropriately powered improvement tests as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

It is aimed at a different outcome, constrained optimization in experiments. No evidence is given that it provides appropriately powered improvement tests; the needed mechanism is to use offline paired evaluation and longer controlled observation before making strong claims at the appropriate boundary.

B:

It emphasizes causal evidence of prompt improvement without closing the stated gap in appropriately powered improvement tests. By contrast, the remedy to use offline paired evaluation and longer controlled observation before making strong claims acts on the condition that makes this scenario fail.

D:

Improving practical significance of a tested improvement does not imply appropriately powered improvement tests. The selected design must use offline paired evaluation and longer controlled observation before making strong claims, whereas this choice omits that direct protection or verification step.

Question 224

A team declares a model winner after checking the dashboard every hour until the p-value dips below its cutoff. Which approach should be recommended?

  1. Set sample size, endpoints and stopping rules before the test.
  2. Control multiple-comparison risk and verify the selected candidate on fresh data.
  3. Use safety guardrails as stopping criteria alongside the primary business metric.
  4. Include effect size and lifecycle expense in the adoption decision.
  5. Stratify A/B results by request difficulty and inspect critical cohorts.

Explanation

Correct Answer

A:

Repeated unplanned significance checks inflate false-positive risk. Predeclare precommitted experiment stopping as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

B:

The relevant trade-off is not primarily reliable multi-variant evaluation. A successful fix needs precommitted experiment stopping; otherwise precommitted experiment stopping remains unverified even after this change.

C:

The control would be justified if constrained optimization in experiments were the limiting factor. In this scenario the decisive target is precommitted experiment stopping; success requires an architecture that will set sample size, endpoints and stopping rules before the test.

D:

The option belongs to the broader work on practical significance of a tested improvement, but it cannot by itself deliver precommitted experiment stopping. A decision on this case needs the team to set sample size, endpoints and stopping rules before the test and verify that outcome.

E:

This shifts attention toward segment-aware model experiment when the relevant decision is about precommitted experiment stopping. Unless the architecture can set sample size, endpoints and stopping rules before the test, the failure described in the stem remains unresolved.

Question 225

A release performs well in a closed test but degrades after rollout to new customer groups. Which change most directly satisfies this requirement?

  1. Randomize eligible traffic and compare predefined business and safety metrics.
  2. Stratify A/B results by request difficulty and inspect critical cohorts.
  3. Include effect size and lifecycle expense in the adoption decision.
  4. Deploy by canary stages with monitoring and rollback triggers.

Explanation

Correct Answer

D:

Incremental exposure limits blast radius while confirming real production behavior. Predeclare safe staged experimentation as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

The proposed action prioritizes causal evidence of prompt improvement. That can coexist with the right solution, but it cannot replace safe staged experimentation: the application must still deploy by canary stages with monitoring and rollback triggers.

B:

The design concern is real but different: segment-aware model experiment. It does not impose the specific requirement for safe staged experimentation; a suitable solution must deploy by canary stages with monitoring and rollback triggers and demonstrate the result.

C:

This addresses practical significance of a tested improvement; however, the problem asks for safe staged experimentation. Without the safeguard to deploy by canary stages with monitoring and rollback triggers, the original failure mode is still possible.

Question 226

An executive compares satisfaction from two months with different user populations to judge a new agent. Which solution best resolves this operational gap?

  1. Randomize eligible traffic and compare predefined business and safety metrics.
  2. Test the components separately or use a planned factorial experiment.
  3. Use matched cohorts or random assignment and account for calendar effects.
  4. Control multiple-comparison risk and verify the selected candidate on fresh data.

Explanation

Correct Answer

C:

A simple before-and-after difference may reflect user mix rather than the intervention. Predeclare control for observational confounding as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

Improving causal evidence of prompt improvement does not imply control for observational confounding. The selected design must use matched cohorts or random assignment and account for calendar effects, whereas this choice omits that direct protection or verification step.

B:

This recommendation targets experimental isolation of interventions at a different point in the workflow. It leaves control for observational confounding outside the design contract and does not require the team to use matched cohorts or random assignment and account for calendar effects.

D:

This would make sense as an additional project activity for reliable multi-variant evaluation. The priority required here is control for observational confounding, so the workflow still needs a control that will use matched cohorts or random assignment and account for calendar effects.

Question 227

A team runs ten simultaneous prompt variants and reports the best without correction or a validation cohort. What design decision would best fit these facts?

  1. Use offline paired evaluation and longer controlled observation before making strong claims.
  2. Set sample size, endpoints and stopping rules before the test.
  3. Test the components separately or use a planned factorial experiment.
  4. Control multiple-comparison risk and verify the selected candidate on fresh data.

Explanation

Correct Answer

D:

Choosing the maximum observed score from many trials invites winner's curse. Predeclare reliable multi-variant evaluation as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

The option belongs to the broader work on appropriately powered improvement tests, but it cannot by itself deliver reliable multi-variant evaluation. A decision on this case needs the team to control multiple-comparison risk and verify the selected candidate on fresh data and verify that outcome.

B:

This shifts attention toward precommitted experiment stopping when the relevant decision is about reliable multi-variant evaluation. Unless the architecture can control multiple-comparison risk and verify the selected candidate on fresh data, the failure described in the stem remains unresolved.

C:

For a workflow whose primary weakness was experimental isolation of interventions, that proposal could be reasonable. Here the unmet obligation is reliable multi-variant evaluation, and that requires the system to control multiple-comparison risk and verify the selected candidate on fresh data.

Question 228

An A/B test produces a statistically visible improvement that saves just seconds but adds substantial maintenance cost. Which intervention most directly addresses the failure?

  1. Use offline paired evaluation and longer controlled observation before making strong claims.
  2. Randomize eligible traffic and compare predefined business and safety metrics.
  3. Set sample size, endpoints and stopping rules before the test.
  4. Include effect size and lifecycle expense in the adoption decision.

Explanation

Correct Answer

D:

Significance alone cannot establish that a technically detectable gain has business value. Predeclare practical significance of a tested improvement as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

That choice would be responsive to appropriately powered improvement tests, but the reported defect is different. It leaves practical significance of a tested improvement unproved because there is no commitment to include effect size and lifecycle expense in the adoption decision.

B:

There is a legitimate use case for causal evidence of prompt improvement, but it does not supply the test or enforcement needed for practical significance of a tested improvement. The design needs to include effect size and lifecycle expense in the adoption decision.

C:

Consider its actual operating scope: precommitted experiment stopping. The requested assurance is practical significance of a tested improvement, which calls for an implementation that can include effect size and lifecycle expense in the adoption decision, not merely for a related improvement.

Question 229

A monitoring assistant interprets a diagnostic tool’s “all clear” status as proof that a service has recovered. The same tool result flags diagnostics as incomplete and lists four hosts that were never probed. What failure remediation best addresses the unsupported conclusion?

  1. Raise generation temperature so the model can better reconcile apparently conflicting diagnostics.
  2. Validate diagnostic completeness and unresolved-host counts before permitting a global recovery claim.
  3. Ignore the unprobed hosts whenever the transport returned HTTP 200.
  4. Replace the coverage information with a more reassuring natural-language summary.

Explanation

Correct Answer

B:

A tool’s optimistic status label does not supersede its coverage indicators. A completeness gate should require all mandatory hosts or communicate a partial finding with explicit unresolved checks.

Incorrect Answers

A:

Sampling variation does not establish whether unprobed machines are healthy. The contradiction is in the evidence contract and must be resolved with deterministic completeness checks.

C:

Transport success only establishes delivery of the response, not completeness of the diagnostic operation. Omitting failed coverage can turn a partial result into a false safety assurance.

D:

Rephrasing cannot supply missing measurements. The user needs either a completed probe set or a clearly bounded partial result that flags unresolved service risk.

Question 230

An extractor outputs wrong currency totals although source line items are accurate. Which design choice best matches the stated constraint?

  1. Make retrieval failure explicit and prevent unsupported fallback generation.
  2. Route uncertain complex cases to a stronger model and verify with a difficulty-stratified benchmark.
  3. Delegate arithmetic to a tested calculation component and check units before generation.
  4. Require retrieval-based claim checking and an explicit insufficient-evidence path.

Explanation

Correct Answer

C:

Exact computations should not be trusted to probabilistic text continuation when tools can verify them. Reproduce the failure while checking deterministic numerical correctness at retrieval, tool and model boundaries, changing one hypothesis at a time.

Incorrect Answers

A:

Although grounding-preserving outage behavior can be valuable, it is not the acceptance condition in the question. The missing provision is deterministic numerical correctness, addressed by the control to delegate arithmetic to a tested calculation component and check units before generation.

B:

It is aimed at a different outcome, capability escalation for hard cases. No evidence is given that it provides deterministic numerical correctness; the needed mechanism is to delegate arithmetic to a tested calculation component and check units before generation at the appropriate boundary.

D:

That is primarily a technique for hallucination control on absent evidence. It lacks the explicit measure for deterministic numerical correctness that is necessary here, namely a system that will delegate arithmetic to a tested calculation component and check units before generation.

Claude Certified Architect - Professional (CCAR-P) Practice Tests