A support agent can produce beautifully written explanations and still fail its task. It may choose the wrong knowledge article, call the correct tool with the wrong account number, or report that a refund was issued when the transaction never succeeded. Evaluating a Microsoft Foundry agent therefore requires more than grading the final paragraph. The evaluation system should measure what the user needed, which evidence the agent used, which actions it attempted, what actually happened and whether the behavior complied with safety and business rules.<\/p>\n
Foundry provides built-in evaluators for output quality, groundedness, safety and agent behavior, including tools-focused measures. These can accelerate assessment, but no single score should be treated as definitive evidence of reliability. Agent evaluations are engineering tests: they need a representative dataset, a carefully defined expected outcome, stable decision thresholds and a mechanism for investigating failures. Before choosing an evaluator, decide what failure would be most damaging in the real workflow.<\/p>\n
Consider an agent that books a technician visit. A successful task might require finding the correct customer, checking entitlement, retrieving available slots, obtaining a confirmed choice and writing a booking to the scheduling service. A fluent message that lists three appointments is not success if none was reserved. The task contract should distinguish informational progress from committed state. It should specify which fields must match the original request and what the agent must do when the customer record is ambiguous.<\/p>\n
Different users and intents demand different evaluation targets. A document-answering assistant may require citations that support each factual claim. A case-management agent may require exact record IDs and an authorized update. A security assistant may need to refrain from recommending disallowed actions even when a retrieved document urges otherwise. A good evaluation suite includes these behaviors as explicit acceptance conditions rather than reducing everything to a generic \u201chelpfulness\u201d score.<\/p>\n
Test cases should also record the environment. The same agent response may be valid when a user has a certain license and invalid when the user’s permissions are narrower. Include entitlements, document versions, available tools, expected refusal conditions and downstream service state as part of each scenario. Without this context, graders can disagree for reasons unrelated to model quality.<\/p>\n
Groundedness asks whether claims follow from the supplied material. Relevance asks whether the answer addresses the actual request. Correctness asks whether a claim or action matches reliable ground truth. These can diverge. An assistant may accurately quote an outdated policy and therefore be grounded in its retrieved passage but still give an answer that is no longer correct. A concise correct answer can be highly relevant even if its writing style is plain. A polished explanation may be ungrounded despite sounding authoritative.<\/p>\n
Build representative ground truth where possible. For a product policy, retain a versioned decision table of eligibility conditions and outcomes. For a retrieval task, record which source documents are expected and which tempting documents should not influence the answer. For a multi-step operation, specify final system state rather than merely the sequence of words an agent should emit. Ground truth should be reviewed by domain owners and updated when business rules change; otherwise evaluation gradually rewards yesterday’s correct behavior.<\/p>\n
Retrieval quality needs its own measures. Did the system return passages covering the user’s relevant facts? Did it filter by authorized access? Was an outdated passage ranked ahead of the currently approved version? An answer-level groundedness score cannot reveal an access-control bug in retrieval. The problem must be caught at the query and document-selection stage before the selected text becomes model context.<\/p>\n
Foundry’s agent evaluators include concepts such as tool selection, tool input accuracy, tool-call success, output utilization and navigation efficiency. These focus on the path from request to outcome. Imagine an agent that updates the correct customer record only after making five unauthorized queries against unrelated records. The final task may be successful, but the process is unacceptable. A process-focused evaluator can reveal wasted calls, incorrect arguments and misuse of information that an output-only judge would miss.<\/p>\n
Tool call accuracy is particularly important when tools have similar names. A search function is not a write function, and a dry-run price estimate must not be mistaken for a committed order. Include negative examples where a tool’s response is partial, contradictory or unavailable. The agent should use only confirmed returned values and state clearly when the system of record cannot verify completion. In higher-stakes tasks, independent application checks should enforce the invariants rather than relying entirely on a model-judged trace.<\/p>\n
Efficiency is more subtle than simply minimizing the number of tool calls. A deliberate verification call may be necessary before a financial change, while repeated searches of the same index may be wasteful. Define optimality around safe task completion, not an arbitrary quota. If a new agent version reduces tool calls by skipping an authorization check, it should fail even if its average latency improves. Metrics must reflect the actual workflow contract.<\/p>\n
Teams often begin with neat benchmark questions, then discover that production users speak in fragments and change their minds mid-task. Include misspelled customer names, multiple matching records, changed permissions, missing attachments, conflicting instructions, old document versions and conversations that span several turns. Create cases in which the correct action is to ask for clarification or stop. An agent that confidently guesses a customer identity should not receive a high score for sounding decisive.<\/p>\n
Segment cases by difficulty, business risk and workflow stage. A low-risk internal FAQ and a high-impact data export should not have interchangeable passing thresholds. Test multiple languages if the service claims to support them. Cover permission boundaries across user roles and tenant contexts. Include harmless but irrelevant content as a distractor so the agent is measured on selecting evidence, not on parroting the last passage it saw.<\/p>\n
Preserve a hidden regression set that developers do not continually tune against. If every evaluator prompt and expected result is visible while changes are made, an agent may appear to improve only because the team has optimized to a narrow suite. Separately maintain exploratory adversarial tests that evolve with new tool integrations. Recorded production incidents can become excellent new test cases after sensitive information has been removed.<\/p>\n
Safety evaluation should test both harmful output and harmful action. A prompt-injection attack might arrive from a retrieved web page asking the agent to email a secret. Even if the model declines to quote the secret in its final response, a tool call that transmitted the information is a serious failure. Evaluate the complete trace and downstream state. A user-input filter alone cannot cover instructions hidden in documents and tool results.<\/p>\n
Access-control and data-exfiltration tests need hard pass\/fail rules. If a test user is not entitled to a payroll file, any retrieval or output that exposes the file’s contents is a violation regardless of the agent’s friendly wording. Those tests should be backed by deterministic authorization checks and audit logs. Evaluator models can help prioritize qualitative concerns, but privileged actions and leakage thresholds need defensible technical enforcement.<\/p>\n
Security reviewers should be able to explain why a scenario failed and identify the control that ought to stop it. The concepts behind scoped authorization<\/a> matter in every agentic system: helpfulness does not confer access rights. Treat policy refusals, human approval and reduced tool permissions as positive behaviors when they are the expected safe outcome.<\/p>\n Some tasks have no single exact correct paragraph. An expert may judge whether an incident summary fairly represents conflicting evidence or whether a troubleshooting explanation would guide an engineer to the correct checks. For those cases, use a rubric with observable criteria: identifies the core cause, distinguishes confirmed facts from hypotheses, names a safe next step, and avoids unsupported certainty. Separate substantive accuracy from style so polished prose does not dominate a technical judgment.<\/p>\n Have more than one reviewer grade a sample and investigate disagreement. If reviewers interpret \u201ccomplete\u201d differently, the rubric is underspecified. Blind comparisons of two agent versions can help reduce brand and presentation bias, but reviewers still need representative task context. For model-based graders, inspect calibration against reviewed examples and rerun checks when the grader model or rubric changes. A changed scoring system should not be reported as evidence that the production agent improved.<\/p>\n Versioning is critical. Record agent instructions, tool schemas, model deployment, dataset version, evaluator configuration and any retrieval-index snapshot necessary to reproduce the test. A score without provenance is difficult to compare to a later score because too many components may have changed at once.<\/p>\n Offline tests find known weaknesses before release, while operational monitoring reveals cases the team did not predict. Trace retrieval and tool calls, measure task outcomes, collect user corrections, watch for escalations and audit errors in downstream systems. Segment metrics by task type and customer group; an overall average may disguise serious failures in a small high-impact workflow. Avoid storing sensitive raw conversations unnecessarily in an analytics system.<\/p>\n Establish deployment gates. A new version must meet minimum task-completion and safety thresholds, cannot introduce critical regressions, and should remain observable during rollout. When feasible, evaluate a candidate in shadow mode before allowing it to perform writes. Gradually increase traffic and keep a rollback path. The principles behind AI platform operations<\/a> become concrete at this stage: evaluation belongs in release engineering, not only in a one-time model-selection exercise.<\/p>\n For a refund agent, the final scorecard should report how many eligible requests were correctly completed, how many ineligible requests were refused, how often tool calls used the correct account and amount, how many cases required human intervention, and how many unauthorized actions were blocked. Quality, efficiency and safety must be read together. A fast agent that occasionally issues an unauthorized payment is not \u201cmostly reliable\u201d in the sense that matters.<\/p>\n Evaluating Foundry agents is ultimately about proving bounded behavior under realistic conditions. Test the information used, the decisions proposed, the actions actually executed and the outcomes confirmed. Treat an attractive aggregate score as an invitation to inspect the underlying cases, not as permission to stop examining the system.<\/p>\n","protected":false},"excerpt":{"rendered":" A support agent can produce beautifully written explanations and still fail its task. It may choose the wrong knowledge article, call the correct tool with […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2925","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.exam-topics.info\/blog\/wp-json\/wp\/v2\/posts\/2925","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.exam-topics.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.exam-topics.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.exam-topics.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.exam-topics.info\/blog\/wp-json\/wp\/v2\/comments?post=2925"}],"version-history":[{"count":0,"href":"https:\/\/www.exam-topics.info\/blog\/wp-json\/wp\/v2\/posts\/2925\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.exam-topics.info\/blog\/wp-json\/wp\/v2\/media?parent=2925"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.exam-topics.info\/blog\/wp-json\/wp\/v2\/categories?post=2925"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.exam-topics.info\/blog\/wp-json\/wp\/v2\/tags?post=2925"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Use human review without turning it into guesswork<\/h3>\n
Connect offline evaluation to live monitoring<\/h3>\n