A chat assistant that handles ten internal test requests can fail spectacularly on its first busy Monday. The prompt is unchanged, the model is unchanged and the retrieval corpus has not grown. What changed is contention: multiple users arrive together, long conversations consume more tokens and a search indexer may be embedding new documents in the same operational window. Capacity planning for AI systems therefore needs more than a price-per-token estimate or a reassuring average response time.
Model quotas matter to Microsoft AI-103 because a successful request under light load says little about behavior during a traffic spike. Model rate limits, embedding workloads, search capacity and media processing have different scaling constraints. A support application needs a deliberate response when one component is saturated: bounded queueing, retry with backoff, an approved fallback or a clear refusal. Measure the end-to-end consequence of throttling rather than treating a model quota as the only capacity number. The Foundry deployment configuration needs a capacity budget that the team can reproduce during a controlled release and under actual traffic.
A request budget is not a user budget
A single user interaction can invoke a model more than once. An agent may classify a request, retrieve documents, reason about a tool response and compose an answer. A multi-agent workflow can fan out into several specialist calls before returning control to its coordinator. Counting only conversations conceals the number of requests, tokens and tool operations that the platform must actually serve.
Build a capacity estimate from the path an interaction follows. How many model calls occur on a normal turn? How much retrieved evidence is placed into prompts? What happens when the model has to retry a tool call? Which requests include a large image or many seconds of audio? Effective Foundry service selection weighs those request patterns against supported deployments, quotas and latency budgets before a model becomes a production dependency.
An internal finance assistant offers a simple illustration. A routine invoice question might involve a brief classification and one grounded answer. A disputed invoice could retrieve policy records, compare transactions and prepare a recommendation for approval. Treating those as equal “one request” events leads to an underestimate precisely where latency and cost matter most. Budget by observed workload classes, not a single global average.
Understand the changing meaning of quota
Foundry and Azure OpenAI limits include concepts such as tokens per minute, requests per minute, concurrent requests and model-specific deployment capacity. The exact scope depends on deployment type, model and whether a particular workload has moved to subscription-level quota management. Microsoft began introducing subscription-level quota management during 2026; older descriptions that imply one permanent per-resource or per-region rule should not be copied into an operational plan.
Capacity figures should be checked for the actual model deployment and region rather than assumed from an example table. A larger quota allocation is not the same as guaranteed low latency, and additional deployed capacity cannot repair slow tool dependencies. Similarly, a queue growing inside a search or extraction service may create user-facing delays even while model request limits remain comfortably below their thresholds.
Maintain a current register of deployments, quota scopes, regions, request classes and escalation options. Include the distinction between platform-enforced rejection and application-enforced throttling. The former is a symptom that capacity has been exceeded; the latter can be a deliberate product policy that protects a system before it reaches an unreliable state.
Draw a latency budget across the whole request
End-to-end latency is the sum of waiting and work across several services, not merely the generation time printed in a model dashboard. Authentication, search, embedding a query, semantic ranking, tool calls, downstream APIs, moderation and generation can each add delay. The largest mean component is not always responsible for poor tail performance: occasional slow retries or one stalled approval service may dominate the slowest five percent of requests.
Give each stage a practical budget and record the distribution, especially p95 or p99 where traffic volume permits useful measurement. If retrieval consumes most of the budget, reducing response length may have little effect. If the tool layer is repeatedly timing out, switching models is a distraction. AI observability and evaluation should expose stage-level latency and failures so teams can distinguish retrieval bottlenecks, tool timeouts and model response delays.
For interactive experiences, design what users see while work proceeds. A reliable progress indication, bounded timeout and clear retry behavior often matter more than shaving a small amount off a single backend call. For batch document processing, throughput, queue age and cost per completed item may be better service measures than individual response latency.
Treat tokens as an architectural decision
Prompt tokens are consumed by system instructions, conversation history, retrieved passages, tool schemas and task-specific input. Completion tokens grow with the model’s output. The cost of a long-running agent is therefore shaped by choices such as keeping every past message, exposing dozens of irrelevant tools or retrieving a large volume of low-value evidence. Those decisions also increase the likelihood of hitting context limits and making inconsistent selections.
Constrain context to the material the task genuinely requires. Summarize old conversation state when the workflow permits it, prune tool definitions that are irrelevant to the current step and use retrieval filters to reduce unsuitable evidence. For a policy agent, returning the two authorized clauses needed to answer a question is often safer and less expensive than providing an entire handbook. RAG and vector search makes this a relevance and authorization problem as well as a cost problem.
Small models may perform well for narrow classification or extraction tasks while a more capable model handles difficult reasoning. A routing design can be economical, but only after it is evaluated against genuine task difficulty. Adding multiple classifiers and fallbacks can cost more than a single well-chosen model if the supposed optimization sends too many requests through extra steps.
Backoff should not turn an incident into a flood
When a service returns a throttling response, immediate synchronized retries often make the situation worse. Use bounded exponential backoff with jitter where appropriate, respect retry guidance from the service and put an upper limit on how many attempts one user action can trigger. A queue or concurrency cap can absorb a short burst, but it should not grow without limit until all useful work is obsolete.
Retries must also consider side effects. A read-only model request can usually be retried more safely than a tool action that issued a purchase order or changed a customer record. For stateful agent operations, reconcile the prior outcome before repeating an external write. Make timeout and idempotency behavior visible in the tool contract rather than leaving each application developer to invent it.
Graceful degradation is a product decision. A travel support assistant might omit a nonessential recommendation when enrichment is delayed, while an account-access agent should refuse to proceed without verified authorization. The failure mode that preserves trust is not always the one that returns an answer fastest. Build service tiers around consequence and user expectations.
Search, extraction and media each have their own bill
It is easy to focus on model inference costs and ignore the expense of preparing the evidence that makes the model useful. Ingestion may involve OCR, layout analysis, chunking, embedding generation, index storage and scheduled updates. Video and image workflows add larger payloads, media storage and, for some models, asynchronous rendering jobs. The total cost per useful answer includes these shared processing and retention costs.
Plan separately for steady-state query volume and periods of reindexing. Changing an embedding strategy can create a large one-time backfill at the same time the application is receiving normal traffic. Running that work at uncontrolled parallelism can exhaust quota or degrade interactive responses. Mark high-impact migrations and backfills in the release plan, and give them limits that protect customer-facing tasks.
Rich-media use cases need concrete limits: maximum accepted duration or size, reasonable output variants, retention windows and cancellation behavior. A user who requests twenty variants of an image is making a different operational request from one who asks a model to label a single uploaded photo. Align feature permissions and budgets to those differences.
Measure unit economics without rewarding bad answers
Cost per request is attractive because it is easy to chart, but cost per successful outcome is more useful. A cheap response that invents a policy exception can create far more downstream work than a slightly more expensive response supported by evidence. Track the share of requests that pass task-specific quality and safety checks, require human correction or end without a useful result. Then compare cost alongside those outcomes.
Use a small collection of realistic traffic profiles for load tests: ordinary turns, long-history users, retrieval-heavy questions, failed-tool recovery, multilingual audio and peak-concurrency bursts. Combine synthetic volume with carefully selected production-like scenarios, while keeping private data out of test logs. Measure rate limiting, queue growth, failure recovery and quality deterioration under load—not just throughput.
For major architecture changes, define a baseline before testing the candidate. Did a smaller model materially reduce cost while preserving accuracy? Did tighter document filtering improve both citation quality and latency? Did streaming reduce perceived wait time without hiding long-running backend work? The results may differ by workload; avoid extrapolating one benchmark to every agent in the portfolio.
Treat observability as capacity management
A production dashboard should connect user requests to the component that made them expensive or slow. Record model deployment, estimated input and output tokens, search latency, tool durations, retries, throttles, safety interventions and release version. Keep sensitive content out of unrestricted telemetry. Aggregations can indicate a trend, but detailed traces are needed to discover why a particular failure occurred.
Set alert thresholds around behavior that needs action: sustained quota rejection, abnormal queue age, missing ingestion updates, unexpected token growth after a prompt release, or a spike in tool-call retries. A high number of requests alone may be a healthy sign of adoption. An alert must distinguish legitimate demand from degraded service to avoid teaching operators to ignore it.
Capacity planning succeeds when an engineer can explain what happens to a real user at the limit. Identify each bottleneck, reserve budget for priority traffic, make retries finite and measure cost per completed acceptable task. A faster or cheaper model request is not an improvement if it increases failed transactions or hides a saturated downstream index.