A supplier-invoice extractor can return valid JSON with a plausible bank account and still be wrong because it merged details from two documents. The schema has done its job—it constrained the fields—but it has not established which supplier owns the account. Before a payment system accepts the result, the application needs source attribution, cross-field consistency checks, and an explicit decision about missing evidence. This difference between representation and acceptance is tested through the architectural reasoning required by Anthropic CCA-F. An unsupported account number should lead to a documented failure or human verification, not a more confident prompt or an endless attempt to make the JSON look complete.
Define acceptance before drafting the prompt
A prompt that asks Claude to extract information accurately is hard to evaluate. Specify the fields, their business meaning, expected source evidence and permitted uncertainty states. An invoice may require legal supplier identity, document number, issue date, currency and amount. Each field should have a rule that a reviewer can explain. For example, the total must come from a supported invoice line or calculation, not from a guess based on another document. When a value is absent, an explicit missing state is better than an invented placeholder.
Criteria should target false positives as well as missing output. If an extractor labels every signature-like mark as a valid approval, a well-formed result can still authorize the wrong person. State what evidence counts and what does not. For a supplier account number, require association with the verified legal entity rather than mere presence somewhere in the file. Such definitions make the validator and the model prompt agree on the intended task instead of relying on informal notions of plausible-looking data.
Separate schema validation from semantic validation
Schema validation answers questions about structure: is the amount numeric, is the currency code properly formed and are mandatory fields present? It cannot establish that the amount was actually printed on the invoice or that the bank account matches the correct supplier. Semantic validation examines relationships between returned fields and authoritative evidence. A schema-valid invoice can still be duplicated, internally inconsistent or bound to the wrong purchase order.
Build acceptance as layers. Parse and validate the declared response shape first, then check business constraints and source grounding. If the invoice currency is EUR but the procurement contract is in USD, preserve the actual invoice currency and separately determine whether conversion is permitted. Do not silently rewrite source facts to satisfy a downstream field expectation. This distinction complements structured-output contracts while focusing on verification after generation rather than response formatting alone.
Design explicit validators for critical fields
A production system should verify document identifiers against expected formats, detect duplicate records and test arithmetic relationships such as subtotal plus tax equaling total within defined tolerances. It may cross-check an invoice’s supplier ID against an authenticated vendor registry. Some checks can be deterministic; others require human interpretation. Classify them so a team knows what is established by code and what remains a model judgment.
A high-value validator records the reason for rejection. A currency mismatch, unknown vendor, missing invoice date and contradictory bank details are not one generic invalid response. Precise failure categories allow the application to choose a suitable next step. They also make evaluation more useful: if most failures arise from unreadable scans, prompt rewriting is unlikely to solve them. The remedy may be higher-quality document capture or a manual exception process.
Use feedback loops when the evidence is sufficient
A repairable classification boundary may improve through contrasting few-shot examples, but no example can manufacture a missing bank account or signature. Repeated failures involving unavailable or contradictory proof should follow a defined escalation rule rather than another model attempt. The validator must measure whether each retry improves grounding, not merely whether it finally produces a value.
Suppose Claude extracts a due date in an invalid format although the scan clearly contains the correct date. A targeted retry can provide the parsing error, required format and original source reference without exposing unrelated information. The model receives actionable feedback and can attempt a corrected response. By contrast, an unreadable date is not repairable simply by insisting that Claude try harder. The workflow should return an unresolved state and request better evidence.
Retries need a clear contract. Keep the relevant input stable, identify the exact defect and limit the number of attempts. Do not repeatedly send a long generic rejection message that invites the model to fabricate a value merely to satisfy validation. After a correction, run every affected check again. A changed date might interact with payment terms, so a local field repair may require recomputing derived values. Feedback improves quality only when the application knows what it is asking to correct.
Do not treat repeated guessing as improvement
An extractor that fails a semantic consistency check five times may be missing a necessary source document or using an inappropriate model strategy. More attempts can produce different plausible answers without better evidence. Define a stopping condition based on error type, retry count and whether the last attempt made measurable progress. A syntax error may justify a correction loop; a conflicting supplier identity may require independent verification or an authorized reviewer.
This protects both cost and correctness. Track the number of retries, their causes and the proportion resulting in genuinely accepted records. A workflow that improves its parse success rate while raising false supplier matches has become more dangerous, not more reliable. Separate valid completion, human-review routing and unresolved source limitations in reporting. The goal is a correct decision process, not maximizing the share of cases that emerge with a non-null answer.
Preserve evidence through each revision
Every generated field should retain enough provenance to show which document or passage supported it. When a retry changes a value, record the old candidate, rejection reason, new candidate and evidence reference. Otherwise a final report may look consistent while hiding a history of conflicting values. Source provenance lets an auditor distinguish a legitimate correction from arbitrary model output drift.
Keep business records and model drafts separate. A failed extraction should not overwrite an already accepted vendor account merely because the latest model response contains a new bank number. Use versioned candidate records and an explicit acceptance gate before committing changes. If a human corrects an ambiguous field, record the reviewer and the original evidence. Such controls make later investigation possible when a transaction is disputed or a source document is updated.
Calibrate confidence to the cost of error
A model may express high confidence in a wrong bank account, particularly when two similar invoices appear together. Do not equate language-model confidence with a calibrated probability of business correctness. Build routing rules from observed validation results, document quality, evidence conflicts and risk severity. A low-value formatting discrepancy might be automatically corrected; a supplier banking change warrants an authenticated verification path regardless of how confidently Claude describes it.
The decision threshold should reflect consequence. Financial transfers, regulated reports and sensitive identity matching demand stronger proof than a product-description draft. Evaluate false approvals, missed legitimate documents and manual-review volume separately. A model that routes most cases to review may be safe but operationally unusable; one that rarely escalates may conceal errors. The architecture needs a deliberate operating point supported by data, not by an untested instruction to be conservative.
Test against hard documents, not just clean examples
A useful evaluation set includes duplicated invoices, different date formats, mixed currencies, combined scans, missing pages, handwritten changes, partial receipts and mismatched supplier accounts. Freeze the source documents and expected business outcomes, then compare the baseline with any revised prompt or validator. Examine each failure case by type so improvements to common documents do not hide regressions on rare but consequential exceptions.
Separate cases used to tune prompts from held-out cases. Otherwise the model may appear better because it has seen each tricky example repeatedly. Record source identity, expected extraction, acceptance rule and whether a human reviewer agreed. Track the complete system’s success, including retrieval, OCR where applicable, structured output, validation and record commit. The model’s response quality is only one component of a reliable extraction pipeline.
Work a two-invoice bank account mismatch
An accounts-payable service receives a PDF containing an invoice for Supplier A followed by a remittance note from Supplier B. Claude returns Supplier A’s name but Supplier B’s bank account. The JSON parser accepts the response, yet the vendor-registry check finds that the account does not match Supplier A. The application rejects the candidate with a precise mismatch reason, preserves both source references and requests an evidence-grounded correction only if the documents contain sufficient information.
If the scan establishes that Supplier A’s bank details are absent, the correct outcome is manual verification rather than another fabricated account number. The downstream payment system receives no accepted bank change until the authorized process resolves the discrepancy. This is the central architectural choice: a model generates hypotheses from messy material, validators establish what can be accepted, feedback repairs recoverable mistakes and escalation handles missing or conflicting truth. The application, not the model’s confidence, decides when the record is ready to act on.