Three engineering teams consume the same Foundry model deployment. One team begins running a new summarization workload and exhausts shared model capacity; another service starts failing requests even though its own traffic never changed. Meanwhile a security review discovers that a model-facing API accepts calls from more application identities than its owners intended. Neither problem is solved by a generic warning in a prompt. The Microsoft SC-500 exam includes configuring AI Gateway and agent guardrails because model access now requires controls familiar from enterprise API security: identity, limits, policy, observation and safe failure.
The gateway boundary is only one part of securing an AI system. AI workload security architecture also covers data, identities, models and tools. The gateway protects selected calls and can apply resource-level policies, while the service behind it must still control what tools may do and what data is permitted. A model request being authenticated, rate-limited and content-filtered is not proof that a resulting business action is authorized.
Draw the real request path
Start by locating the client application, identity provider, API Management instance, Foundry resource and project, deployed model and any downstream tool. Label each hop with its authentication method and permitted audience. With a configured Foundry AI Gateway, Azure API Management sits between clients and Foundry model endpoints so applicable policy can be enforced centrally. If an application can still call an ungoverned direct model endpoint, the perimeter may be incomplete. Verify which routes are actually used rather than assuming all clients honor the chosen architecture diagram.
For a customer-support application, require the front-end to communicate through its backend rather than exposing privileged model credentials to a browser. The backend obtains the appropriate credential, addresses the gateway and handles failures without retry storms. Record separate control owners for Entra access, API Management policies, model deployment configuration and application authorization. These owners may be on different teams, making audit trails and change responsibilities essential to keeping a shared gateway maintainable over time.
Token limits and quotas protect different outcomes
A token-per-minute limit constrains consumption over a short interval, while a total quota limits aggregate use over a specified longer window. The Foundry Control Plane documentation differentiates these constraints and their outcomes: an exceeded rate limit may produce an HTTP 429, while an exceeded total quota can produce an HTTP 403. Those are materially different from authentication rejection, application authorization denial or a model safety refusal. A client should not treat them all as the same retryable fault.
Build capacity policy around measured workloads. A small internal assistant may be able to wait or defer some requests; a contact-center tool that helps a human resolve an urgent case may need reserved throughput and priority handling. Set project- or consumer-level limits based on legitimate demand and test how the service behaves at the boundary. Sentinel collection and diagnostics can help correlate the operational symptom with identity and service events, but the gateway should emit enough non-sensitive request metadata for owners to diagnose a rejection without collecting an entire confidential prompt.
Authenticate upstream callers and downstream backends separately
A gateway’s managed identity may authorize it to reach the Foundry backend, while client applications need their own validated identities or credentials at the gateway edge. Conflating those trust relationships can turn a shared service into an unintended privilege amplifier. Scope each client’s authorization to the model or operation it needs. Review who can edit API Management policies, because policies that use the gateway’s managed identity may themselves become sensitive controls; overly broad policy-edit access can undermine downstream authentication assumptions.
For a negative test, present a valid token intended for another audience and confirm the gateway rejects it. Then use an approved caller against an unauthorized model deployment. Finally try reaching the model directly through an unapproved path and verify the organization’s intended network or access control prevents the bypass. Application identities and consent distinguish issuance of a token from each service’s obligation to authorize the requested operation.
Use policies to make intended behavior measurable
Rate limits, header transformations, back-end routing and request inspection can be useful, but a large unreviewed policy expression can be difficult to maintain. Write the expected user and service outcomes first, then implement the narrowest supported API Management policy sequence. For example, a valid application principal may send supported text content to a specific model but should not invoke a different deployment or consume a whole team quota. Configure telemetry that can report policy name, result and request correlation without logging credentials or private source documents.
Test policy changes with a small known workload and a forbidden request. Also verify how the gateway reports timeouts, unavailable backends and model throttling. A blanket catch-all retry rule may increase costs and hide the real failure. Treat quotas as an availability and cost safeguard, not as a sufficient security program. Underlying prompts, retrieval and tool execution can still expose data after a request has passed every gateway capacity rule.
Foundry agent guardrails are not tool authorization
Prompt injection can come from user messages, retrieved documents or tool results. A retrieved page might instruct the agent to send a secret to an unrelated endpoint. Model or agent guardrails can help detect or constrain unsafe text patterns, but they should be considered alongside source trust, output validation and deterministic tool permissions. The business API that could transfer funds or modify accounts must still enforce entitlement and transaction policy. The agent should not be able to remove that requirement by producing a persuasive explanation in natural language.
For a test, provide an agent a malicious-but-plausible document that asks it to bypass an approval. The desired result is that the tool call is absent or denied, the user’s workflow is handled safely and a useful diagnostic signal exists. Trace why it was blocked: model-layer safety policy, agent instruction, tool schema, API authorization or a human-approval gate. A robust design does not rely on a single fragile layer. Agent identity and Purview data posture adds the complementary task of ensuring an agent does not access inappropriate source records in the first place.
Monitor model safety with the right supporting controls
The SC-500 blueprint includes Defender for AI Service in Defender for Cloud, as well as AI-related security dashboards and Microsoft Foundry protection features. Configure supported protection and monitoring for the actual workload and plan, then determine who investigates findings. A policy can detect or flag risk without necessarily blocking every malicious action; distinguish observation, prevention, investigation and enforcement in reporting. Avoid claiming a preview service or a particular portal screen is available in every tenant when the current licensing, region or deployment mode varies.
An alert about unusual AI activity should lead to a correlation path: caller identity, gateway policy, model deployment, retrieved context, tool action and business outcome. Do not copy raw conversation content into many monitoring systems simply to obtain correlation. A threat or safety signal only becomes actionable when the team can identify which boundary failed and whom to notify. The goal is an evidence-driven response, not a high count of security products on a dashboard.
Design recovery behavior before production launch
When the model deployment is unavailable, the application may need to present a clear service delay, use an approved alternative or fall back to a deterministic workflow. It should not silently route regulated data to a model in another region or provider without policy review. Rate limiting and failover rules should respect data-location, business-separation and authorization requirements. Similarly, when the gateway itself fails, developers should not revert to insecure direct service credentials merely to restore the demo.
Create a deployment procedure that records gateway policy versions, model versions, region, API authentication, quotas, exception owners and rollback instructions. Test a new policy against legitimate traffic and known denials before promoting it. A safe release should preserve both customer experience and the security boundary. Application and API perimeter controls address related upstream web and backend protections that belong outside the model-call gateway.
An important design check is what happens when a request passes the gateway but the back-end model or agent makes an unacceptable decision. Policy telemetry should distinguish an upstream authorization success from a downstream application validation failure. For example, the gateway might record a properly authenticated call within quota even though a CRM tool later denies the agent’s attempted change to a protected account. Preserve enough correlation to explain both outcomes without exposing private prompts to every API administrator. Review these cases together with business owners because a high rate of downstream denials could reveal a workflow bug, a privilege problem or intentionally blocked misuse. Treating them as generic model errors would conceal the control that actually protected the transaction.
Reconcile cost, safety and legitimate service availability
A shared gateway can be efficient when teams agree on resource ownership and measurement. Without those agreements, a central administrator may set caps so strict that approved users cannot work, or exceptions so broad that a heavy application can starve everyone else. Use successful-task rate, capacity, denied requests, model latency and validated safety outcomes to tune the deployment. Investigate whether an apparent quota problem comes from excessive prompt context, an agent loop, real demand growth or a configuration mistake before raising all limits.
Review a model request at four separate boundaries: whether this caller is authorized, whether the selected model deployment is permitted, whether the request stays within resource and safety policies, and whether any resulting business action is approved. The controlling service may be Entra, API Management, Foundry, the agent or the downstream application. Prove the boundary with one permitted and one blocked transaction instead of relying on a general statement that AI Gateway is enabled.