AzureFeeds

Authors: Caroline Yeh, Ali Mahmoudzadeh, Morteza Ziyadi

 If you’ve shipped an agent to production, or you’re close, you’ve already made a quiet bet: that the evals you ran during development reflect what happens when real users interact with the real system. That bet is harder to win than it looks. Not because evaluation is unsolved, but because agents break in ways single-turn metrics can’t see. A task silently stalls mid-session. A factual claim slips through ungrounded. A conversation scores 4.2 on coherence while the user abandons it on turn three. None of that shows up as an error. None of it registers in a per-turn score. And if the test harness generating those scores has its own calibration problems, like a simulator that coaches your agent, a trace sampler that keeps pulling the same three request types, or an evaluator with a 20% flip rate, then you’re not measuring your agent’s quality. You’re measuring your eval stack’s optimism. 

This post is for the engineers and applied scientists responsible for that bet. The ones writing the eval pipelines, choosing the judge models, deciding which traces go into the fine-tuning corpus, and fielding the question “is this agent ready to ship?” on every release cycle. 

Over the past two weeks, the Microsoft Foundry observability team published five research studies on how to measure agents and, just as importantly, how to measure the tools doing the measuring. Today, we’re pulling them into one prescriptive path. 

Before you ship your agent, make sure you have: 

1. Multi-turn evaluated: session-level quality scored across the whole conversation, not averaged across turns 


2. A defined rubric: what “good” means for your task, your agent, your success criteria, not a generic catalog entry 


3. Simulated users: test conversations that behave like real users, scored with a validated rubric 


4. Sampled production traces:  the right traces for breadth of coverage, not just the most frequent ones 


5. Benchmarked: a standardized, reproducible yardstick for your deployment or agent 

These aren’t a one-time gate. They’re a loop you re-run on every meaningful prompt, model, or tool change. Here’s what each step does, why it matters to the engineer on the hook for agent quality, and where to go deep. 

Why evaluation is harder for agents than it was for models 

If you came up evaluating LLMs on fixed benchmarks or RAG pipelines on retrieval metrics, agents require a shift in how you think about measurement and it’s worth naming explicitly before the five steps. 

With a model, the unit of analysis is a response. With an agent, the unit of analysis is a session: a multi-turn conversation in which the agent asks clarifying questions, calls tools, retries when something fails, and stitches the whole thing into an outcome. A session can have every individual response score well on relevance, fluency, and faithfulness and still be unfinished, off-policy, or quietly hallucinated. Averaging turn-level scores doesn’t recover those properties, and it can actively mislead you and the session that failed your user reads as a pass. 

This creates a second-order problem specific to ML engineers: any tool you use to score your agent is itself a system that can be wrong. A weak evaluator that always outputs “pass” will flatter your agent. A judge with high run-to-run variance will generate phantom regressions on every rerun. A simulator that coaches your agent when it misses a step hides the exact failures you’re trying to measure. If you don’t instrument the evaluation stack with the same rigor you’d apply to the agent, you end up with a measurement system that’s less reliable than your intuition. 

The implication is straightforward: evaluation is recursive. You have to measure the evaluator, not just the agent. Every study below does exactly that; accuracy, reliability, and cross-judge robustness, so you know which evaluators to trust as hard gates, which to use as triage signals, and where the eval stack itself needs tuning before you can trust what it’s telling you. 

Step 1  Multi-turn evaluate 

Four session-level evaluators: Task Completion, Customer Satisfaction, Groundedness, and Conversation Coherence, each targeting a distinct property of the whole conversation. 

Start here because it defines the problem every other step is helping you solve: you need to score the session, not the turn. Each of the four evaluators targets a property that can only be assessed across the full conversation, and the design is intentionally complementary, an agent can pass one and fail another, and that gap is diagnostic, not redundant. 

Task Completion measures if the user’s task is successfully and completely accomplished by the end of the session. 

Customer Satisfaction shows how satisfied would the user be with the agent’s performance across the session. 

Groundedness checks if the agent’s factual claims across all turns are supported by the conversation’s grounding sources. 

Conversation Coherence verifies that the conversation flows logically across turns; state maintained, no contradictions, sensible progression. 

The study validated all four with benchmark datasets, paired validity and reliability metrics across four independent runs, and six judge models. Two headline findings for the engineer making deployment decisions: 

Task Completion and Customer Satisfaction reach strong enough agreement with ground truth and low enough run-to-run variance to use as single-pass scores in release gating so that you don’t need four reruns to trust the verdict. 

Groundedness is the exception. It’s the hardest of the four, and the wrong tool for a hard release gate. Smaller judges don’t just shift the mean, they collapse the hallucinated cluster in the score distribution, missing the minority-class failures the metric exists to catch. Deploy it as an aggregate trend signal with a frontier reasoning judge, route low-scoring sessions to human review, and track it over time. 

Run these evaluators on your test set before each release. Gate on Task Completion and CSAT. Trend on Groundedness. Use the gaps between evaluators as the diagnostic signal. 

Read the study Evaluating Multi-Turn Agents: A Quality Study of Microsoft Foundry’s Multi-Turn Evaluators | Microsoft Community Hub 

Step 2  Define the rubric 

Auto-generated, task-specific rubric evaluators that turn “what good looks like for your agent” into a weighted score with per-dimension explanations which is adaptive, context-aware, and reusable across development iterations. 

This is where most evaluation stacks fall short, and where the opportunity is greatest. Generic evaluators measure generic properties. Task Completion tells you whether the job got done. CSAT tells you whether the user was satisfied. But neither captures the success criteria that are specific to your agent, your domain, and your policies. 

“Verify the customer’s identity before issuing a refund.” “Confirm the alert is active before paging the on-call engineer.” “Flag deviations from the contract template before drafting negotiation language.” These aren’t properties you can buy off the shelf. They live in your system prompt, your task definition, your policy documentation. A generic rubric can’t see them. A fixed catalog can’t anticipate them. And without them, your evaluations are measuring something real but just not the thing your users actually care about. 

The auto-generated rubric evaluator is designed to close exactly this gap. Give it your agent definition and examples — the context you already have — and it generates a task-specific rubric with weighted dimensions and per-case explanations you can reuse across iterations. Unlike fixed multi-turn evaluators that apply the same criteria to every agent, the rubric adapts to what your agent is actually supposed to do. Change your agent’s task scope, update its policy, expand its tool set. Generate a new rubric, and your evaluation pipeline stays current without manual re-authoring. 

In validation, generated rubrics aligned with trusted reference signals at the case level (scores on individual samples), and ranked candidate agents consistently with ground-truth oracles (correlation of agents rankings between grounf-truth and evaluations). Not only the discriminative quality of the generated rubrics was high, but also a large fraction of expert-authored rubrics are automatically generated. Rubrics show high reliability, reruns produce consistent verdicts and good separability, the evaluator’s ability to tell one candidate agent from another is decisive.

This is the step that makes your evaluation specific rather than generic, adaptive rather than fixed, and grounded in your agent’s actual task rather than a one-size catalog. Run multi-turn evaluators to know whether the session succeeded. Run the rubric to know whether your agent did its specific job correctly and to understand the dimensions where it fell short. 

Read the study: Auto-Generated Rubric Evaluators: Building Context-Aware Evaluators for AI Agents | Microsoft Community Hub 

Step 3  Simulate users 

USR-8: eight orthogonal metrics for scoring a user simulator that separate behavioral correctness from prose quality — so you can trust the test conversations generating your scores. 

User simulators let you push an agent through hundreds of plausible conversations between every code push, without waiting for real users. The catch, which matters directly to anyone who owns the eval pipeline: a simulator that sounds polished can quietly inflate your agent’s scores; a simulator that coaches your agent when it misses a step hides the regressions you’re trying to catch. Neither failure mode shows up in the agent’s score sheet. 

USR-8 is an eight-metric rubric specifically for scoring user-simulator output, designed around a distinction that matters to the engineer, not just the researcher: behavior and style are separate axes. Clarity, relevance, steering (the no-coaching guard rail), responsiveness, consistency, and persona fidelity describe what the simulator does. Realism describes how it sounds. They move independently. Across 1,200 scored conversations in three domains, airline customer service, SRE incident triage, and legal contract review, the Foundry simulator scored at or near ceiling on every behavioral metric; realism was the one gap, and a focused prompt revision closed it with no regression elsewhere. 

The key design decision worth internalizing as an eval engineer: most of what makes a simulator good or bad lives in the prompt policy, not the orchestration code. A prompt swap between two harnesses closed most of the measured gap on every per-conversation metric. If your simulator is underperforming, rewrite the prompt before you replace the framework. 

Read the study How to Score a User Simulator: Introducing USR-8 | Microsoft Community Hub 

Step 4  Sample production traces 

MinHash farthest-first diversity sampling: server-side, zero extra per-token cost, selecting for coverage of the input space rather than a mirror of production frequencies. 

Once your agent is in production, you have more traces than you can evaluate and the subset you pick shapes everything downstream: what your eval catches, what your fine-tuning learns, what your rubric generation sees. The default instinct is uniform random sampling. It’s statistically unbiased and mirrors your production distribution accurately. It’s also dominated by the handful of common request patterns that make up the bulk of your traffic, and systematically under-represents the rare prompts, unusual tool-call sequences, and edge cases that are the most important inputs for both evaluation and fine-tuning. 

Intelligent sampling uses MinHash signatures and farthest-first traversal to select a subset that covers as much of the input space as possible, including the long tail a uniform sample buries. It runs server-side with no LLM or embedding calls — zero additional per-token cost, completing in under a minute on typical trace pools. On the primary validation dataset (5,000 WildChat traces, 100 selected), diversity sampling produced higher lexical diversity and larger vocabulary than a uniform-random baseline. An LLM judge preferred the diversity-sampled subset most of the time for training and evaluation use. Fine-tuning on a diversity-sampled corpus converged faster with comparable held-out generation quality. 

Reach for diversity sampling when breadth of behavior drives downstream quality — evaluation suites, rubric generation, fine-tuning corpora. Reach for a uniform sample when you need faithful production-frequency representation — latency profiling, error-rate estimation, distribution-level SLA checks. 

Read the study Intelligent sampling in Microsoft Foundry: the science behind selecting better production traces | Microsoft Community Hub 

Step 5  Benchmark 

Standardized open-source benchmarks run against your own model deployment or agent — with your judge model, your configuration, and side-by-side comparison across runs. 

Public leaderboard scores tell you how a model performs in general. They don’t tell you how your deployment performs on the reasoning, math, or domain-knowledge questions that matter for your application or whether it regressed after a prompt or model-version change. That’s a different measurement, and it requires running the benchmark yourself, against your own deployment, under the same conditions you plan to compare. 

Benchmarks in Microsoft Foundry (preview) make that a first-class part of the development workflow. You select from a catalog of well-known suites, reasoning, math, science, and truthfulness (GPQA Diamond, BIG-Bench Hard, MuSR, TruthfulQA, BBEH, AIME 2025, and more), each with its own curated dataset, task category, and built-in scorer. Run them from the portal wizard or the REST API, compare results side by side in the evaluation group view, and pin a stable judge model and benchmark version so the score is a reproducible yardstick you can trust across releases — not a one-time number tied to a single configuration. 

The practical pattern for release gating: pick one or two reasoning-heavy benchmarks (GPQA Diamond or MuSR are good defaults), target your agent, run it on every meaningful change to the underlying model, system prompt, or tool set, and treat the delta as the signal — not the absolute score. 

Read the study Benchmarks in Microsoft Foundry (preview): Standardized model and agent quality checks | Microsoft Community Hub 

The thread running through all five 

The key design decision across these studies: we measured every evaluator the way we ask you to measure your agent. Benchmark datasets. Paired validity and reliability metrics. Multiple judge models per evaluator so we could separate “evaluator works” from “this particular judge works.” That methodology produced a result that’s more useful than a single accuracy number: for each evaluator, you now know which axis to trust it on, which judge tier it requires, and whether the caveats live in the score quality or only in the threshold placement. 

That distinction matters operationally. CSAT with a smaller judge produces good scores at a miscalibrated threshold, a fixable problem that can be solved by re-tuning the cutoff on your own data. Groundedness with a smaller judge produces genuinely weaker scores, not a fixable problem, a judge-replacement decision. Accuracy tables look the same. The per-judge ablation is what tells you which category of caveat you’re dealing with. 

What makes this approach different 

• Session-level measurement, not turn-level averages. The four multi-turn evaluators measure properties that only exist across a full conversation. No amount of single-turn metric aggregation recovers task completion or cross-turn grounding. 

• Task-specific rubrics, not one-size-fits-all criteria. The auto-generated rubric evaluator builds evaluation criteria from your agent’s own context — adaptive to your task, your domain, and your policies — rather than applying a fixed catalog to every agent regardless of what it actually does. 

• Validated evaluators, not just provided ones. Every evaluator above ships with a published quality study — accuracy, reliability, and cross-judge robustness — so you know where to trust it as a hard gate and where to treat it as a triage signal before you wire it into your CI pipeline. 

• Coverage over representativeness, by design. Diversity sampling deliberately surfaces the long tail your uniform sample buries — the rare inputs and edge cases that evaluation and fine-tuning need most and random sampling systematically skips. 

• Simulator quality is a first-class metric. USR-8 gives you a rubric for the test harness itself. If the simulator isn’t measured, the scores it produces aren’t trustworthy — and you won’t know in which direction they’re wrong. 

• One workflow, linked to the traces. Multi-turn evaluation, rubric generation, simulation, trace sampling, and benchmarking live in the same observability surface, against the same traces — not five disconnected tools requiring five separate data pipelines. 

Getting started 

Start by reading our Evaluating AI Agents Practical Gide with Microsoft Foundry for holistic view and approach.  Then run the path in order, keeping it as a loop on every meaningful change.   

• Start building: ai.azure.com 

• Watch the Build session (BRK252): aka.ms/build26-BRK252  

• Read the Build recap: From observability to ROI for AI agents on any framework 

• Join the community: aka.ms/ai/discord 

The teams shipping agents that hold up in production aren’t the ones running the most evals. They’re the ones who know which evals to trust, which to treat as signals, and which parts of their measurement stack to fix before they can believe any of the numbers. These five steps don’t guarantee a perfect agent. They help ensure that when something breaks, the failure shows up in your data — not in your users’ experience. 

That’s what evaluation is actually for.