Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 22

Question 210

A live system changes retrieval, prompts and model versions simultaneously before comparing results. What design decision would best fit these facts?

  1. Build format-specific golden cases with known expected fields and extraction failures.
  2. Pin test fixtures and vary one component at a time for diagnostic comparisons.
  3. Use rubric or semantic correctness checks while preserving exact assertions for critical fields.
  4. Create independent held-out cases and enforce test-set separation.
  5. De-identify or secure test fixtures and restrict access under data-handling policy.

Explanation

Correct Answer

B:

Controlled evaluation isolates which modification caused improvement or regression. Include held-out and high-consequence cases for reproducible component-level evaluation, and keep evaluation fixtures separate from prompt demonstrations.

Incorrect Answers

A:

The control would be justified if modality-specific regression tests were the limiting factor. In this scenario the decisive target is reproducible component-level evaluation; success requires an architecture that will pin test fixtures and vary one component at a time for diagnostic comparisons.

C:

This shifts attention toward appropriate grading for variable language when the relevant decision is about reproducible component-level evaluation. Unless the architecture can pin test fixtures and vary one component at a time for diagnostic comparisons, the failure described in the stem remains unresolved.

D:

For a workflow whose primary weakness was independent evaluation evidence, that proposal could be reasonable. Here the unmet obligation is reproducible component-level evaluation, and that requires the system to pin test fixtures and vary one component at a time for diagnostic comparisons.

E:

Although privacy-preserving test data governance can be valuable, it is not the acceptance condition in the question. The missing provision is reproducible component-level evaluation, addressed by the control to pin test fixtures and vary one component at a time for diagnostic comparisons.

Question 211

An evaluator rewards persuasive but weakly grounded answers despite a written rubric. Its citation check also accepts fabricated references that merely look like valid document IDs. Which TWO evaluation improvements directly address these issues? Choose TWO.

  1. Create independent held-out cases and enforce test-set separation.
  2. Resolve each cited identifier to a retrieved source span and score whether that span actually supports the associated claim.
  3. Grade citation formatting and response fluency while allowing source identifiers to remain unverified strings.
  4. Calibrate the judge with human-labeled pairs and explicit evidence-based rubric.
  5. Evaluate tool traces and side effects in addition to the final response.

Explanation

Correct Answers

B:

A citation string is not proof of grounding. Linking claims to exact evidence spans and checking entailment exposes fabricated or irrelevant references that a fluency-focused judge would overlook.

D:

A fluency-biased judge can assign high scores to unsupported answers. Human-labeled comparison pairs and an evidence-focused rubric calibrate the evaluator against the failure that matters; verifying each reference against a source span is a separate hard check.

Incorrect Answers

A:

This addresses independent evaluation evidence; however, the problem asks for validated evaluator reliability and claim-to-source entailment verification. Without the safeguard to calibrate the judge with human-labeled pairs and explicit evidence-based rubric, the original failure mode is still possible.

C:

Valid-looking citation syntax is not evidence of an actual source or a supported claim. This would preserve the current evaluation weakness instead of establishing source-span verification.

E:

The alternative could reduce risk associated with trace-aware agent evaluation, yet the described behavior requires validated evaluator reliability and claim-to-source entailment verification. That outcome depends on the capability to calibrate the judge with human-labeled pairs and explicit evidence-based rubric.

Question 212

A multimodal pipeline ingests scans, spreadsheets and emails, but tests include only plain-text PDFs. Which design choice best matches the stated constraint?

  1. Build format-specific golden cases with known expected fields and extraction failures.
  2. Add stratified high-risk examples and report class-specific recall.
  3. Pin test fixtures and vary one component at a time for diagnostic comparisons.
  4. Include freshness, source-version and citation-verification assertions in the golden set.

Explanation

Correct Answer

A:

Input representation changes upstream error modes; one document format cannot certify all modalities. Include held-out and high-consequence cases for modality-specific regression tests, and keep evaluation fixtures separate from prompt demonstrations.

Incorrect Answers

B:

This would make sense as an additional project activity for rare-case evaluation coverage. The priority required here is modality-specific regression tests, so the workflow still needs a control that will build format-specific golden cases with known expected fields and extraction failures.

C:

By itself, this improves reproducible component-level evaluation, but the case is not asking the architect to optimize that dimension. The requirement is modality-specific regression tests, supported by an approach to build format-specific golden cases with known expected fields and extraction failures.

D:

Its main effect concerns source-version correctness in tests. The case explicitly hinges on modality-specific regression tests; to resolve it reliably, the implementation has to build format-specific golden cases with known expected fields and extraction failures, which this choice does not establish.

Question 213

A policy assistant returns nondeterministic phrasings and exact-string comparisons mark valid answers wrong. Which architecture adjustment is most appropriate?

  1. Evaluate tool traces and side effects in addition to the final response.
  2. Use rubric or semantic correctness checks while preserving exact assertions for critical fields.
  3. De-identify or secure test fixtures and restrict access under data-handling policy.
  4. Include freshness, source-version and citation-verification assertions in the golden set.

Explanation

Correct Answer

B:

Evaluation should tolerate harmless language variation but enforce the required factual content. Include held-out and high-consequence cases for appropriate grading for variable language, and keep evaluation fixtures separate from prompt demonstrations.

Incorrect Answers

A:

For a workflow whose primary weakness was trace-aware agent evaluation, that proposal could be reasonable. Here the unmet obligation is appropriate grading for variable language, and that requires the system to use rubric or semantic correctness checks while preserving exact assertions for critical fields.

C:

It is aimed at a different outcome, privacy-preserving test data governance. No evidence is given that it provides appropriate grading for variable language; the needed mechanism is to use rubric or semantic correctness checks while preserving exact assertions for critical fields at the appropriate boundary.

D:

It emphasizes source-version correctness in tests without closing the stated gap in appropriate grading for variable language. By contrast, the remedy to use rubric or semantic correctness checks while preserving exact assertions for critical fields acts on the condition that makes this scenario fail.

Question 214

An agent's final answer looks correct although it accessed a tool outside approved policy. Which next step is most defensible?

  1. Evaluate precision, recall and review workload at the operating threshold.
  2. Evaluate tool traces and side effects in addition to the final response.
  3. Use rubric or semantic correctness checks while preserving exact assertions for critical fields.
  4. Include freshness, source-version and citation-verification assertions in the golden set.

Explanation

Correct Answer

B:

Behavioral correctness includes authorized actions; output-only scoring misses unsafe paths. Include held-out and high-consequence cases for trace-aware agent evaluation, and keep evaluation fixtures separate from prompt demonstrations.

Incorrect Answers

A:

Consider its actual operating scope: threshold-aware evaluation under workload constraints. The requested assurance is trace-aware agent evaluation, which calls for an implementation that can evaluate tool traces and side effects in addition to the final response, not merely for a related improvement.

C:

The proposed measure is aimed at appropriate grading for variable language, not trace-aware agent evaluation. The architecture must instead evaluate tool traces and side effects in addition to the final response; a separate change is still needed for trace-aware agent evaluation.

D:

The relevant trade-off is not primarily source-version correctness in tests. A successful fix needs trace-aware agent evaluation; otherwise trace-aware agent evaluation remains unverified even after this change.

Question 215

The evaluation dataset contains customer IDs and confidential files in an unprotected shared repository. Which approach should be recommended?

  1. Create independent held-out cases and enforce test-set separation.
  2. De-identify or secure test fixtures and restrict access under data-handling policy.
  3. Pin test fixtures and vary one component at a time for diagnostic comparisons.
  4. Use rubric or semantic correctness checks while preserving exact assertions for critical fields.

Explanation

Correct Answer

B:

Evaluation quality does not justify uncontrolled replication of sensitive production records. Include held-out and high-consequence cases for privacy-preserving test data governance, and keep evaluation fixtures separate from prompt demonstrations.

Incorrect Answers

A:

Its main effect concerns independent evaluation evidence. The case explicitly hinges on privacy-preserving test data governance; to resolve it reliably, the implementation has to de-identify or secure test fixtures and restrict access under data-handling policy, which this choice does not establish.

C:

The proposed action prioritizes reproducible component-level evaluation. That can coexist with the right solution, but it cannot replace privacy-preserving test data governance: the application must still de-identify or secure test fixtures and restrict access under data-handling policy.

D:

The design concern is real but different: appropriate grading for variable language. It does not impose the specific requirement for privacy-preserving test data governance; a suitable solution must de-identify or secure test fixtures and restrict access under data-handling policy and demonstrate the result.

Question 216

A retrieval pipeline passes summary tests but often cites an older version of the source. What is the best production design response?

  1. Include freshness, source-version and citation-verification assertions in the golden set.
  2. Use rubric or semantic correctness checks while preserving exact assertions for critical fields.
  3. Create independent held-out cases and enforce test-set separation.
  4. Build format-specific golden cases with known expected fields and extraction failures.

Explanation

Correct Answer

A:

Grounding evaluation must check document validity, not just semantic similarity. Include held-out and high-consequence cases for source-version correctness in tests, and keep evaluation fixtures separate from prompt demonstrations.

Incorrect Answers

B:

That is primarily a technique for appropriate grading for variable language. It lacks the explicit measure for source-version correctness in tests that is necessary here, namely a system that will include freshness, source-version and citation-verification assertions in the golden set.

C:

Improving independent evaluation evidence does not imply source-version correctness in tests. The selected design must include freshness, source-version and citation-verification assertions in the golden set, whereas this choice omits that direct protection or verification step.

D:

This recommendation targets modality-specific regression tests at a different point in the workflow. It leaves source-version correctness in tests outside the design contract and does not require the team to include freshness, source-version and citation-verification assertions in the golden set.

Question 217

A developer repeatedly tweaks prompts until a public benchmark score rises by one point. Which change most directly satisfies this requirement?

  1. Pin test fixtures and vary one component at a time for diagnostic comparisons.
  2. Evaluate precision, recall and review workload at the operating threshold.
  3. Create independent held-out cases and enforce test-set separation.
  4. Build format-specific golden cases with known expected fields and extraction failures.
  5. Keep a frozen final holdout and separate exploratory development cases.

Explanation

Correct Answer

E:

Repeated optimization against one benchmark overfits the measurement instrument. Include held-out and high-consequence cases for defensible final generalization estimate, and keep evaluation fixtures separate from prompt demonstrations.

Incorrect Answers

A:

The relevant trade-off is not primarily reproducible component-level evaluation. A successful fix needs defensible final generalization estimate; otherwise defensible final generalization estimate remains unverified even after this change.

B:

The control would be justified if threshold-aware evaluation under workload constraints were the limiting factor. In this scenario the decisive target is defensible final generalization estimate; success requires an architecture that will keep a frozen final holdout and separate exploratory development cases.

C:

The option belongs to the broader work on independent evaluation evidence, but it cannot by itself deliver defensible final generalization estimate. A decision on this case needs the team to keep a frozen final holdout and separate exploratory development cases and verify that outcome.

D:

This shifts attention toward modality-specific regression tests when the relevant decision is about defensible final generalization estimate. Unless the architecture can keep a frozen final holdout and separate exploratory development cases, the failure described in the stem remains unresolved.

Question 218

A fraud alert assistant's false positives cause human queues to overflow, but only recall is reported. What design decision would best fit these facts?

  1. Include freshness, source-version and citation-verification assertions in the golden set.
  2. Evaluate precision, recall and review workload at the operating threshold.
  3. Evaluate tool traces and side effects in addition to the final response.
  4. Add stratified high-risk examples and report class-specific recall.

Explanation

Correct Answer

B:

A system must balance missed incidents against capacity consumed by benign escalations. Include held-out and high-consequence cases for threshold-aware evaluation under workload constraints, and keep evaluation fixtures separate from prompt demonstrations.

Incorrect Answers

A:

The design concern is real but different: source-version correctness in tests. It does not impose the specific requirement for threshold-aware evaluation under workload constraints; a suitable solution must evaluate precision, recall and review workload at the operating threshold and demonstrate the result.

C:

That choice would be responsive to trace-aware agent evaluation, but the reported defect is different. It leaves threshold-aware evaluation under workload constraints unproved because there is no commitment to evaluate precision, recall and review workload at the operating threshold.

D:

There is a legitimate use case for rare-case evaluation coverage, but it does not supply the test or enforcement needed for threshold-aware evaluation under workload constraints. The design needs to evaluate precision, recall and review workload at the operating threshold.

Question 219

A new assistant prompt performs better in a demo, but the organization needs proof before broad rollout. Which intervention most directly addresses the failure?

  1. Deploy by canary stages with monitoring and rollback triggers.
  2. Use matched cohorts or random assignment and account for calendar effects.
  3. Randomize eligible traffic and compare predefined business and safety metrics.
  4. Use safety guardrails as stopping criteria alongside the primary business metric.

Explanation

Correct Answer

C:

Controlled experiments reduce seasonal and user-mix confounding versus anecdotal side-by-side impressions. Predeclare causal evidence of prompt improvement as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

This recommendation targets safe staged experimentation at a different point in the workflow. It leaves causal evidence of prompt improvement outside the design contract and does not require the team to randomize eligible traffic and compare predefined business and safety metrics.

B:

This is a valid architectural concern when optimizing control for observational confounding. Here it leaves causal evidence of prompt improvement unaddressed, whereas the intended change must randomize eligible traffic and compare predefined business and safety metrics before the case can be considered resolved.

D:

By itself, this improves constrained optimization in experiments, but the case is not asking the architect to optimize that dimension. The requirement is causal evidence of prompt improvement, supported by an approach to randomize eligible traffic and compare predefined business and safety metrics.

Question 220

A proposed model change cuts costs but may affect only complex customer inquiries. What should the solution architect prioritize?

  1. Test the components separately or use a planned factorial experiment.
  2. Set sample size, endpoints and stopping rules before the test.
  3. Stratify A/B results by request difficulty and inspect critical cohorts.
  4. Deploy by canary stages with monitoring and rollback triggers.

Explanation

Correct Answer

C:

Average savings may hide degraded service for the tasks that most need advanced reasoning. Predeclare segment-aware model experiment as an endpoint or guardrail, then use an appropriate comparison design to control confounding and multiple looks.

Incorrect Answers

A:

This shifts attention toward experimental isolation of interventions when the relevant decision is about segment-aware model experiment. Unless the architecture can stratify A/B results by request difficulty and inspect critical cohorts, the failure described in the stem remains unresolved.

B:

For a workflow whose primary weakness was precommitted experiment stopping, that proposal could be reasonable. Here the unmet obligation is segment-aware model experiment, and that requires the system to stratify A/B results by request difficulty and inspect critical cohorts.

D:

It is aimed at a different outcome, safe staged experimentation. No evidence is given that it provides segment-aware model experiment; the needed mechanism is to stratify A/B results by request difficulty and inspect critical cohorts at the appropriate boundary.

Claude Certified Architect - Professional (CCAR-P) Practice Tests