Question 200
A document extractor meets average accuracy targets but fails disproportionately on handwritten forms. Which solution best resolves this operational gap?
- Measure critical-field recall and high-severity omission rates, not satisfaction alone.
- Segment precision and recall by document format and risk cohort.
- Publish risk-weighted metrics plus breakdowns by task class and severity.
- Use terminology accuracy and human review of high-risk clauses as primary metrics.
Explanation
Correct Answer
B:Aggregate scores hide systematic coverage gaps that matter to the deployment population. Set a threshold for performance on hard input subgroups on the real task population and count failures and high-severity misses in its denominator.
Incorrect Answers
A:The proposed measure is aimed at coverage of mandatory legal facts, not performance on hard input subgroups. The architecture must instead segment precision and recall by document format and risk cohort; a separate change is still needed for performance on hard input subgroups.
C:The control would be justified if risk-sensitive interpretation of aggregate metrics were the limiting factor. In this scenario the decisive target is performance on hard input subgroups; success requires an architecture that will segment precision and recall by document format and risk cohort.
D:The option belongs to the broader work on domain-specific translation fidelity, but it cannot by itself deliver performance on hard input subgroups. A decision on this case needs the team to segment precision and recall by document format and risk cohort and verify that outcome.
Question 201
A recommendation assistant sometimes reveals restricted records in otherwise useful answers. What design decision would best fit these facts?
- Publish risk-weighted metrics plus breakdowns by task class and severity.
- Track groundedness and citation-to-source support separately from style quality.
- Define safety leakage rate and severity-weighted policy violations as release gates.
- Measure critical-field recall and high-severity omission rates, not satisfaction alone.
Explanation
Correct Answer
C:An overall quality score cannot trade away hard confidentiality constraints. Set a threshold for safety-specific acceptance thresholds on the real task population and count failures and high-severity misses in its denominator.
Incorrect Answers
A:The proposed action prioritizes risk-sensitive interpretation of aggregate metrics. That can coexist with the right solution, but it cannot replace safety-specific acceptance thresholds: the application must still define safety leakage rate and severity-weighted policy violations as release gates.
B:The design concern is real but different: evidence-backed response accuracy. It does not impose the specific requirement for safety-specific acceptance thresholds; a suitable solution must define safety leakage rate and severity-weighted policy violations as release gates and demonstrate the result.
D:That choice would be responsive to coverage of mandatory legal facts, but the reported defect is different. It leaves safety-specific acceptance thresholds unproved because there is no commitment to define safety leakage rate and severity-weighted policy violations as release gates.
Question 202
A compliance reviewer reports 96% accuracy after excluding every case where its model abstained. Investigators find that difficult files are much more likely to be excluded. Which metric design best prevents this misleading performance claim?
- Keep accuracy on answered cases as the sole release metric, because abstentions have no incorrect answer.
- Increase the model response-length limit and assume fewer abstentions will raise accuracy.
- Report accuracy together with response coverage and error rates within the abstained/high-risk subset.
- Publish the satisfaction score of employees who received a completed answer.
Explanation
Correct Answer
C:Conditional accuracy alone hides selection bias: difficult items leave the scored sample when the model abstains. Coverage and independent error assessment for deferred cases make the residual risk visible.
Incorrect Answers
A:Abstention can be appropriate, but excluding deferred cases changes the denominator and can conceal the highest-risk failures. A reliable release gate must also report coverage and the disposition of cases sent to review.
B:A longer response does not establish that missing evidence can be recovered. It can instead encourage unsupported conclusions while leaving selection bias in the quality report untouched.
D:Satisfaction among served users excludes the deferred population and cannot demonstrate that hard compliance files were safely resolved. The evaluation needs a denominator covering all eligible cases.
Question 203
A product assistant invents customer-friendly details when the correct answer is unavailable. What should the solution architect prioritize?
- Measure end-to-end completion, escalation and timeout rates together with answer quality.
- Measure end-to-end p95 latency with stage-level attribution.
- Define safety leakage rate and severity-weighted policy violations as release gates.
- Measure appropriate abstention alongside supported-answer precision.
- Track groundedness and citation-to-source support separately from style quality.
Explanation
Correct Answer
D:A safer system may decline to answer unsupported queries rather than optimize raw answer coverage. Set a threshold for calibrated abstention quality on the real task population and count failures and high-severity misses in its denominator.
Incorrect Answers
A:The option belongs to the broader work on reliable complete task outcomes, but it cannot by itself deliver calibrated abstention quality. A decision on this case needs the team to measure appropriate abstention alongside supported-answer precision and verify that outcome.
B:This shifts attention toward end-to-end user latency when the relevant decision is about calibrated abstention quality. Unless the architecture can measure appropriate abstention alongside supported-answer precision, the failure described in the stem remains unresolved.
C:For a workflow whose primary weakness was safety-specific acceptance thresholds, that proposal could be reasonable. Here the unmet obligation is calibrated abstention quality, and that requires the system to measure appropriate abstention alongside supported-answer precision.
E:This addresses evidence-backed response accuracy; however, the problem asks for calibrated abstention quality. Without the safeguard to measure appropriate abstention alongside supported-answer precision, the original failure mode is still possible.
Question 204
A translation tool is tested only by generic text similarity although exact contract terminology is mandatory. Which architecture adjustment is most appropriate?
- Track groundedness and citation-to-source support separately from style quality.
- Publish risk-weighted metrics plus breakdowns by task class and severity.
- Measure critical-field recall and high-severity omission rates, not satisfaction alone.
- Use terminology accuracy and human review of high-risk clauses as primary metrics.
Explanation
Correct Answer
D:A paraphrase can be fluent yet legally wrong when a defined term changes. Set a threshold for domain-specific translation fidelity on the real task population and count failures and high-severity misses in its denominator.
Incorrect Answers
A:That choice would be responsive to evidence-backed response accuracy, but the reported defect is different. It leaves domain-specific translation fidelity unproved because there is no commitment to use terminology accuracy and human review of high-risk clauses as primary metrics.
B:There is a legitimate use case for risk-sensitive interpretation of aggregate metrics, but it does not supply the test or enforcement needed for domain-specific translation fidelity. The design needs to use terminology accuracy and human review of high-risk clauses as primary metrics.
C:Consider its actual operating scope: coverage of mandatory legal facts. The requested assurance is domain-specific translation fidelity, which calls for an implementation that can use terminology accuracy and human review of high-risk clauses as primary metrics, not merely for a related improvement.
Question 205
A multi-agent workflow appears accurate on completed jobs but silently fails 15% of tasks. Which next step is most defensible?
- Measure end-to-end completion, escalation and timeout rates together with answer quality.
- Measure appropriate abstention alongside supported-answer precision.
- Measure end-to-end p95 latency with stage-level attribution.
- Measure verified first-contact resolution and downstream reopen rates.
Explanation
Correct Answer
A:Quality among survivors conceals unreliable execution; denominator design must include failed runs. Set a threshold for reliable complete task outcomes on the real task population and count failures and high-severity misses in its denominator.
Incorrect Answers
B:By itself, this improves calibrated abstention quality, but the case is not asking the architect to optimize that dimension. The requirement is reliable complete task outcomes, supported by an approach to measure end-to-end completion, escalation and timeout rates together with answer quality.
C:Its main effect concerns end-to-end user latency. The case explicitly hinges on reliable complete task outcomes; to resolve it reliably, the implementation has to measure end-to-end completion, escalation and timeout rates together with answer quality, which this choice does not establish.
D:The option might improve actual issue resolution; that remains insufficient for reliable complete task outcomes. The case is resolved by reliable complete task outcomes, which depends on a design that will measure end-to-end completion, escalation and timeout rates together with answer quality.
Question 206
An executive wants a single performance number across trivial questions and high-loss compliance decisions. Which approach should be recommended?
- Publish risk-weighted metrics plus breakdowns by task class and severity.
- Measure critical-field recall and high-severity omission rates, not satisfaction alone.
- Segment precision and recall by document format and risk cohort.
- Define safety leakage rate and severity-weighted policy violations as release gates.
Explanation
Correct Answer
A:A composite can summarize trends, but critical failures require separate visibility and thresholds. Set a threshold for risk-sensitive interpretation of aggregate metrics on the real task population and count failures and high-severity misses in its denominator.
Incorrect Answers
B:It is aimed at a different outcome, coverage of mandatory legal facts. No evidence is given that it provides risk-sensitive interpretation of aggregate metrics; the needed mechanism is to publish risk-weighted metrics plus breakdowns by task class and severity at the appropriate boundary.
C:It emphasizes performance on hard input subgroups without closing the stated gap in risk-sensitive interpretation of aggregate metrics. By contrast, the remedy to publish risk-weighted metrics plus breakdowns by task class and severity acts on the condition that makes this scenario fail.
D:That is primarily a technique for safety-specific acceptance thresholds. It lacks the explicit measure for risk-sensitive interpretation of aggregate metrics that is necessary here, namely a system that will publish risk-weighted metrics plus breakdowns by task class and severity.
Question 207
A team reports lower model latency even though retrieval now makes the user wait longer. What is the best production design response?
- Measure end-to-end completion, escalation and timeout rates together with answer quality.
- Measure end-to-end p95 latency with stage-level attribution.
- Measure critical-field recall and high-severity omission rates, not satisfaction alone.
- Measure verified first-contact resolution and downstream reopen rates.
Explanation
Correct Answer
B:Users experience the whole workflow; model-only timing is an incomplete service metric. Set a threshold for end-to-end user latency on the real task population and count failures and high-severity misses in its denominator.
Incorrect Answers
A:The alternative could reduce risk associated with reliable complete task outcomes, yet the described behavior requires end-to-end user latency. That outcome depends on the capability to measure end-to-end p95 latency with stage-level attribution.
C:The relevant trade-off is not primarily coverage of mandatory legal facts. A successful fix needs end-to-end user latency; otherwise end-to-end user latency remains unverified even after this change.
D:The control would be justified if actual issue resolution were the limiting factor. In this scenario the decisive target is end-to-end user latency; success requires an architecture that will measure end-to-end p95 latency with stage-level attribution.
Question 208
A classifier has 98% accuracy on common FAQ tickets but untested performance on rare security incidents. Which change most directly satisfies this requirement?
- Evaluate precision, recall and review workload at the operating threshold.
- Include freshness, source-version and citation-verification assertions in the golden set.
- Create independent held-out cases and enforce test-set separation.
- Add stratified high-risk examples and report class-specific recall.
Explanation
Correct Answer
D:Representative coverage includes consequential edge cases, not just the most frequent intent. Include held-out and high-consequence cases for rare-case evaluation coverage, and keep evaluation fixtures separate from prompt demonstrations.
Incorrect Answers
A:The option might improve threshold-aware evaluation under workload constraints; that remains insufficient for rare-case evaluation coverage. The case is resolved by rare-case evaluation coverage, which depends on a design that will add stratified high-risk examples and report class-specific recall.
B:The proposed action prioritizes source-version correctness in tests. That can coexist with the right solution, but it cannot replace rare-case evaluation coverage: the application must still add stratified high-risk examples and report class-specific recall.
C:The design concern is real but different: independent evaluation evidence. It does not impose the specific requirement for rare-case evaluation coverage; a suitable solution must add stratified high-risk examples and report class-specific recall and demonstrate the result.
Question 209
A document-summarization test set uses the same examples shown inside the prompt template. Which solution best resolves this operational gap?
- Create independent held-out cases and enforce test-set separation.
- Include freshness, source-version and citation-verification assertions in the golden set.
- Add stratified high-risk examples and report class-specific recall.
- De-identify or secure test fixtures and restrict access under data-handling policy.
Explanation
Correct Answer
A:Leakage creates inflated confidence and does not demonstrate generalization to new cases. Include held-out and high-consequence cases for independent evaluation evidence, and keep evaluation fixtures separate from prompt demonstrations.
Incorrect Answers
B:Improving source-version correctness in tests does not imply independent evaluation evidence. The selected design must create independent held-out cases and enforce test-set separation, whereas this choice omits that direct protection or verification step.
C:This recommendation targets rare-case evaluation coverage at a different point in the workflow. It leaves independent evaluation evidence outside the design contract and does not require the team to create independent held-out cases and enforce test-set separation.
D:This is a valid architectural concern when optimizing privacy-preserving test data governance. Here it leaves independent evaluation evidence unaddressed, whereas the intended change must create independent held-out cases and enforce test-set separation before the case can be considered resolved.
Question 210
A live system changes retrieval, prompts and model versions simultaneously before comparing results. What design decision would best fit these facts?
- Build format-specific golden cases with known expected fields and extraction failures.
- Pin test fixtures and vary one component at a time for diagnostic comparisons.
- Use rubric or semantic correctness checks while preserving exact assertions for critical fields.
- Create independent held-out cases and enforce test-set separation.
- De-identify or secure test fixtures and restrict access under data-handling policy.
Explanation
Correct Answer
B:Controlled evaluation isolates which modification caused improvement or regression. Include held-out and high-consequence cases for reproducible component-level evaluation, and keep evaluation fixtures separate from prompt demonstrations.
Incorrect Answers
A:The control would be justified if modality-specific regression tests were the limiting factor. In this scenario the decisive target is reproducible component-level evaluation; success requires an architecture that will pin test fixtures and vary one component at a time for diagnostic comparisons.
C:This shifts attention toward appropriate grading for variable language when the relevant decision is about reproducible component-level evaluation. Unless the architecture can pin test fixtures and vary one component at a time for diagnostic comparisons, the failure described in the stem remains unresolved.
D:For a workflow whose primary weakness was independent evaluation evidence, that proposal could be reasonable. Here the unmet obligation is reproducible component-level evaluation, and that requires the system to pin test fixtures and vary one component at a time for diagnostic comparisons.
E:Although privacy-preserving test data governance can be valuable, it is not the acceptance condition in the question. The missing provision is reproducible component-level evaluation, addressed by the control to pin test fixtures and vary one component at a time for diagnostic comparisons.