A travel-expense agent correctly reads a receipt, identifies the employee and proposes an amount. Its answer looks excellent in chat. Yet the submission flow creates two expense lines because a remote API timed out after accepting the first one. A model-quality score will not discover this defect. Nor will a test that mocks the final approval service and checks only that a sentence is well written. The failure lives in the relationship between reasoning, business state and external execution.<\/p>\n
An agent’s fluent response does not prove that an expense transaction happened exactly once or that the right approver authorized it. End-to-end tests must inspect business state, identity boundaries, tool trajectories and recovery after partial failure as well as language quality. Microsoft AB-100<\/a> includes these cross-application validation decisions: a successful demonstration is not evidence of a reliable business workflow until its side effects have been checked.<\/p>\n An invariant is something the business requires to remain true even when the conversation changes. An expense may never be posted twice, a refund must not exceed the approved amount, and an account representative must never see a competitor’s private record. These requirements are better starting points than a list of ideal model answers. They allow test authors to distinguish an acceptable variation in wording from a release-blocking failure.<\/p>\n For each process, write a few representative paths from intake to final business state. An ordinary travel claim, an unreadable receipt, a claim filed after the spending deadline and a duplicate submission exercise different safeguards. Specify the expected records, permissions, approvals and evidence for each. Do not make all test cases \u201chappy path with different wording\u201d; an agent frequently fails at the points where data is missing or a system makes an unexpected transition.<\/p>\n An agentic-first architecture<\/a> separates interpretation, retrieval, business actions and state ownership into explicit contracts. Each should be tested independently before the complete orchestration is tested: responses must use authorized evidence, while actions must leave the intended transactional state without unauthorized side effects.<\/p>\n A scenario ledger should contain a business goal, initial state, authenticated identity, input evidence, relevant policy version, expected tool calls and final state. Include a correlation identifier that follows requests across Copilot Studio, Dynamics 365 and any Foundry services used for model inference or agents. Without that trace, an evaluator may see a failed result and have no way to determine whether the root cause was retrieval, permissions, model output or a downstream API.<\/p>\n Business-state fixtures need deliberate versioning. A sales opportunity that has not yet closed leads to a different recommendation from an identical-looking opportunity marked closed. A customer case with an existing refund must not create another credit. Capture those fields in test setup and verify them afterward. A model answer that says \u201cthe case is resolved\u201d does not prove the authoritative case record changed.<\/p>\n Where real data cannot be used, construct representative, permissioned fixtures with meaningful relationships instead of random standalone strings. A customer record should link to a plausible order, service case and payment state so cross-application tools are exercised in realistic conditions. Retain expected outcomes as data, not just screenshots of past agent conversations that might become invalid after a legitimate platform upgrade.<\/p>\n A retrieval test asks whether the correct source was available, fresh, relevant and permitted to the caller. Include two similarly named documents, a recently revised policy and a restricted record that the user may not access. A success case should show the correct evidence; a negative case should demonstrate that protected content never reached the model. Redacting the generated final answer is too late if unauthorized material was retrieved earlier.<\/p>\n Identity propagation must survive handoffs. If a Teams user asks a customer-service agent for a permitted summary, that agent must not accidentally invoke an ERP integration as a platform administrator with unrestricted scope. Test access decisions at the actual service boundary, including expired sessions, changed roles and a caller who is allowed to read a record but not modify it. These are deterministic assertions, not matters of the model’s opinion.<\/p>\n The source model in grounding and knowledge sources<\/a> informs freshness and provenance tests. Security requirements from AI governance and security<\/a> add denial and audit conditions. Keep the two results separate: a document may be highly relevant but not authorized, or authorized but obsolete. Both failures should block a high-impact decision.<\/p>\n Many agent workflows can produce the right summary through an unsafe route. A purchasing assistant may retrieve a supplier list, call an overprivileged update tool and then produce a polite message saying no change was made. Evaluation should inspect which tools were selected, which arguments were sent, the identity under which they ran and the state they produced. Trace assertions can reveal a dangerous intermediate action even when the final sentence is harmless.<\/p>\n Tests need a controlled answer rubric for qualitative behavior such as acknowledging uncertainty, using evidence and asking for clarification. Combine those judgments with objective checks: required JSON fields, correct entity identifiers, no duplicate write operations, approved action sequence and compliance with explicit amount limits. Automated model-based grading can help triage results but should not be the only authority for high-value financial or identity changes.<\/p>\n Multi-agent workflows introduce failure modes that individual worker tests cannot detect. Multi-agent solution design<\/a> should preserve the inventory record and price version passed between specialists, with idempotent handoffs and a defined response to worker timeouts. Test an orchestrator that combines a correct stock estimate with a stale price to ensure the final decision is rejected or escalated.<\/p>\n Inject a delay after a remote service accepts a transaction but before the caller receives its response. The workflow should reconcile the operation identifier and actual business status instead of blindly reissuing the write. Introduce a search timeout, a revoked connector, a missing approval recipient, an invalid currency field and a model deployment that rejects a request. Record whether each failure leads to safe termination, clarification or a controlled retry.<\/p>\n Not every fallback is allowed to degrade. A customer-information assistant might offer an approved general policy answer when account data cannot be reached, making clear that it has not checked a particular case. A finance approval agent should not replace missing evidence with an inferred amount. These scenarios test the business contract, not just whether the agent maintains a friendly tone during an outage.<\/p>\n Deterministic checks are most useful at the boundary between orchestration and execution. A Copilot Studio agent flow<\/a> can validate an order identifier and reject a forbidden update even if the agent’s prompt suggests proceeding. Test both the route chosen by the orchestrator and the independent rejection inside the flow.<\/p>\n Models, prompts, connectors, retrieved data, agent versions and business APIs change at different speeds. A release may update only a prompt but unintentionally change which tool it chooses; an index refresh may alter the cited evidence; a connector update may rename an output field. Maintain a compatibility record and run the affected scenarios before production traffic changes. Treat source-data schema migrations as possible agent behavior changes even if no prompt was edited.<\/p>\n Run fast deterministic tests early in the pipeline, followed by integration tests and a carefully chosen behavioral suite. The full regression set can be larger, but choose a mandatory release gate around known risks: unauthorized access, double submission, an unavailable approval service, wrong currency and missing source evidence. Those cases should never be waived because a new model performs better on generic question answering.<\/p>\n A release suite should identify compatible versions of the agent, prompt, model, tool schema and grounding index. ALM for agents and models<\/a> becomes meaningful when rollback tests prove those versions can be restored together; reverting only a prompt while leaving an incompatible API cannot recover the business workflow.<\/p>\n A test set represents known situations, not every way users will phrase a request. Production monitoring should surface new error classes, missing knowledge, repeated fallback, incorrect tool selection and human corrections. Review incidents with the business owner to determine whether they reveal a new requirement or a defect in an existing control. Add the high-value lessons to regression cases so the same class of failure is less likely to recur.<\/p>\nBegin from a business invariant rather than a transcript<\/h2>\n
Create a traceable scenario ledger across systems<\/h2>\n
Test the knowledge and identity boundary independently<\/h2>\n
Measure trajectories and actions as well as final answers<\/h2>\n
Engineer failures instead of hoping they are rare<\/h2>\n
Turn platform updates into controlled regression decisions<\/h2>\n
Close the loop with production evidence and human review<\/h2>\n