A retailer introduces an AI service to classify customer complaints, extract an order identifier, search policy guidance and suggest a remedy. The same expensive model handles every step because it was easiest to connect during a pilot. When volume triples, cost rises sharply and simple classifications compete with high-value exceptions for limited capacity. Replacing that one model is not the answer. The architecture needs a decision about which task deserves which capability, and the ability to verify that decision continuously.
Model routing should begin with task requirements and policy boundaries, not an assumption that one powerful model belongs behind every request. Microsoft Foundry’s router can select from eligible underlying models according to its supported configuration, but the application still owns acceptable error rates, schema expectations and data restrictions. Microsoft AB-100 examines how those decisions influence quality and total cost per completed outcome, including whether application-controlled routing offers better control for a given workload.
Classify work before assigning a model
Different stages of a business process tolerate different mistakes. A document classifier might return a small set of labels that a deterministic validator can check. A case summary may allow some stylistic variation but must preserve dates and names. A proposal to issue credit has monetary consequences and requires authoritative evidence plus approval. Put these tasks into classes with explicitly defined accuracy, latency, data-handling and escalation expectations before evaluating models.
For each class, identify the minimum useful context and tools. A short language classification should not receive the entire customer timeline, all enterprise search results and an inventory of transaction tools. This wastes tokens and broadens the chance of disclosing sensitive information. Conversely, a policy exception needs the current rule and applicable customer facts; a cheap model that ignores exceptions may increase operating cost through corrections. Model choice follows the evidence requirement, not a ranking of brand names.
A workload matrix can use columns for risk class, expected call volume, typical token usage, language coverage, latency target, data boundary and validation method. For a multilingual contact center, language fidelity may dominate nominal inference price. For an overnight forecasting explanation, a longer response time may be acceptable. This matrix makes disagreements reviewable and gives the procurement team a reason to support a limited set of deployments rather than a model for every feature.
Understand the router as one layer of a controlled system
Microsoft Foundry model router is presented as a deployable model entry point, with the router selecting among eligible models. Documentation describes routing profiles, subsets, deployment conditions and diagnostic metadata, although capabilities and regional support depend on the deployed version. The architect should validate those conditions rather than claiming that every Foundry model or region is automatically covered. A model that requires separate deployment, for example, may not be an eligible destination until the proper service configuration exists.
Routing does not remove the need to manage outputs. An agent that calls an ERP API expects a structured contract. If two eligible models differ in how reliably they populate required fields, the application must validate the result and decide whether to retry, request clarification or stop. Model-router telemetry can show which model served a request, but a successful response code is not proof that a business action was correct.
Where geography or policy restricts data processing, the approved deployment and routing subset must reflect those limits. A broad fallback policy that silently sends a sensitive request outside an approved data zone can violate the business requirement even if the answer improves. Record which model families are permitted for each workload, what happens when none is available and who can authorize a temporary substitution. Keep this guardrail in configuration and backend controls, not only in the user prompt.
Build a prompt library around purpose and provenance
A prompt library becomes difficult to govern if files are named “final-prompt-v9-new.” Each reusable component needs a business owner, purpose, permitted data types, expected response schema, tool list, source dependencies, version and evaluation suite. For a sales-summary prompt, include how missing opportunity values should be handled, whether the output must cite a CRM record and how the system should behave if the account does not belong to the user.
Separate durable instruction from transaction context. A system-level instruction may define the role and allowed output format. A retrieved policy excerpt is evidence with its own trust level, not a place to insert new instructions. A request from a user expresses intent but does not authorize a financial write by itself. This separation is essential when prompts are reused across teams, because a seemingly harmless revision can change what the agent treats as authoritative.
Make examples meaningful but restrained. Two carefully chosen negative cases can be more valuable than twenty stylistic demonstrations: a document that contains an instruction to ignore access control, and a customer record with conflicting currency amounts. The prompt should guide the model to use evidence and express uncertainty; the application should enforce access and numeric validations independently. Grounding and knowledge architecture explains why reliable evidence must exist before prompt polishing has any value.
Connect prompts to evaluation and a release decision
Treat a prompt revision like application logic. Changing wording can alter selected tools, output format, refusal behavior and how much evidence a model quotes. Record the reason for a change and run it against a fixed set of realistic cases. Test ordinary requests, missing data, conflicting sources, an attempted instruction embedded in a document and a backend tool that returns a partial failure. Compare not just response fluency but correct task completion and forbidden actions.
Use deterministic checks where possible: JSON schema validity, required field preservation, exact authorization outcomes and source freshness. Add sampled human review for tasks where the correct judgment is less mechanical, such as whether a suggested response adequately addresses a disputed billing complaint. A model judge may help triage cases, but it should not become the sole evidence for a high-impact deployment. End-to-end business agent testing shows how prompt checks fit within full workflow validation.
Store the prompt version alongside model identity, routing decision and tool contract version in traces. If customer escalations rise after a release, investigators should know whether a prompt changed, the serving model changed or a data source became stale. That connection is more useful than a dashboard showing aggregate token counts. It also allows a controlled rollback that restores a known compatible set rather than switching arbitrary parameters until symptoms fade.
Model economic value at the level of a completed task
A low cost per token is not the same as a low cost per resolved case. A weak classifier may produce extra handoffs, while a capable model can reduce rework. A router that incurs several failed model attempts may erase its nominal price advantage. Calculate total cost across inference, retrieval, tool calls, evaluations, storage, retries and human correction, then relate that to successful outcomes. The enterprise decision should optimize the process, not an isolated model meter.
Create traffic profiles before committing to a design: ordinary single-turn queries, retrieval-heavy exceptions, long customer conversations, peak-hour bursts and an unavailable backend. Measure p95 latency, failure recovery, routing distribution and time spent waiting for a human decision. Use these profiles to select routing priorities, quotas and graceful-degradation paths. The model mix can be adjusted as evidence accumulates, but a change in the mix must still pass quality and compliance review.
Compare an automatic model-router deployment with explicit application-side classification and routing. The former may reduce operational complexity for supported workloads; the latter gives direct control over policy routing and model-specific guardrails but increases engineering responsibility. Build-versus-buy analysis offers a framework for weighing these options, including vendor dependencies and the cost of future replacement.
Make errors and fallback paths intelligible to the business
A routing service may reject a request because a deployment is unavailable, the prompt exceeds a limit or no eligible model meets a constraint. These are distinct from a valid model answer that lacks enough evidence. Define user-visible and operational behavior for each. A time-critical support assistant might provide a narrower scripted answer during a capacity incident; an agent reviewing financial exceptions should stop rather than silently relax its evidence requirements.
Retries need bounds and state awareness. If a model has only proposed a response, a retry may be relatively safe. If the prior call triggered an external write, repeating the instruction without checking transaction state can duplicate a refund or status change. The tool adapter should distinguish “attempt accepted,” “business action committed” and “outcome unknown.” Model switching must not change those execution rules.
Observability should connect routing decisions to business consequences. Track fallback causes, serving model, p95 response time, quality failures, user correction rate and cost per completed outcome. Agent telemetry can expose a rising cost trend before it becomes a budget crisis and reveal when a cheaper route has shifted too much work to human reviewers.
Control the library as a product over time
A model and prompt registry needs an owner who can deprecate entries. Mark unsupported model versions, record migration deadlines and tell dependent teams when a connector or response schema is changing. A prompt that was validated against a former model should not automatically be described as approved for a replacement model. Keep a compatibility record and a regression suite that identifies behavior changes before production traffic is switched.
Review shared assets periodically for prompt injection, overbroad tool descriptions, sensitive data leakage and stale examples. A successful prompt in a low-risk internal assistant might be unsuitable in a customer-facing agent with account-modification privileges. Create separate approval paths based on the impact of failure and the data involved, so teams can still innovate without weakening control.
The AI Center of Excellence should assign funding and ownership for shared routing infrastructure, approve data-policy constraints and review cost and quality evidence after launch. Model selection is then an accountable business-service decision, not a standing assumption that the most powerful model is always best.