AzureFeeds

1.Why Observability Changes in Agentic AI Systems

A developer recently recounted how a single runaway AI agent loop burned through $500 in OpenAI API charges in just 45 minutes with no errors or alerts raised anywhere in the monitoring stack. That anecdote is not an edge case; as Microsoft’s own observability team notes, “the hard part isn’t getting the agent to work it’s keeping it working,” because models get updated, prompts get tweaked, retrieval pipelines drift, and real user traffic surfaces edge cases that never appeared in any evaluation suite. These realities highlight that modern AI applications are no longer deterministic request-response pipelines. They are agentic AI systems (autonomous, multi-step agents that can plan, reason, invoke tools, and adapt in real time), which fundamentally changes what must be observed. Traditional observability built for servers and microservices cannot tell you whether an AI agent’s output is correct, safe, or cost-efficient. A response with HTTP 200 OK could still be factually wrong or policy-violating, and that gap would never appear on a conventional dashboard.

The question for enterprises has moved from “Can we build it?” to “Can we trust it?” Production-ready AI must not only perform but also behave responsibly, requiring continuous visibility into what agents produce, how they handle edge cases, and whether they remain safe, fair, and compliant. Achieving that trust demands a new generation of observability — one that captures an agent’s thought processes, decision quality, and compliance posture, not just its infrastructure health.

2. From Traditional Observability to AI‑Native Observability

Traditional monitoring assumes relatively predictable execution: a fixed set of endpoints, deterministic paths, and well-understood failure modes. AI agents break this model completely. The key differences driving the need for AI-native observability are:

• Non-Deterministic Execution Paths: The same input can produce wildly different chains of tool calls, retries, and reasoning steps. There is no fixed set of endpoints to monitor.


• Hidden Cost Accumulation: A single user query might trigger 15 LLM calls across different models. Without per-request cost tracking, the bill remains a black box.


• Cascading Failures That Look Like Success: An agent can return a confident-sounding wrong answer after silently failing to retrieve the correct context. The HTTP status says 200; the user says the answer is wrong.


• Extreme Latency Variance: The same query might complete in 800ms or 30 seconds depending on the reasoning path. Traditional p99 alerting does not capture this variability.


• Token Budget Blowouts: One runaway agent loop can burn through a monthly token budget in hours.

Because of these characteristics, AI-native observability extends traditional pillars with semantic and quality insights: what the agent reasoned, how it used context and tools, and whether its outputs meet quality and compliance standards. Problems like context drift, tool misuse, and inefficient reasoning go unnoticed with conventional logging alone.

3. Using One AI Agent to Observe Another

A powerful emerging pattern is employing AI-powered evaluators to monitor and assess other AI agents. Microsoft Foundry implements this directly: its evaluators are functions that assess an agent’s responses, and some evaluators use AI models as judges (an approach known as LLM-as-judge, where a language model scores another model’s output against a rubric), while others use rules or algorithms.

Reasoning Trace Analysis

Foundry’s observability layer captures every stage of an agent’s execution   prompts, tool invocations, tool responses, and output generation — enabling developers to analyze decision paths, identify latency drivers, and pinpoint failure points. Azure Monitor’s AI-Tailored Trace View renders each decision as a readable story: plan → reasoning → tool calls → guardrail checks, allowing teams to identify slow or unsafe steps without sifting through thousands of spans. An AI-Aware Trace Search further allows filtering across millions of runs using GenAI-specific attributes (model ID, grounding score, or cost) to quickly diagnose anomalies

Grounding and Hallucination Detection

Foundry provides text similarity evaluators that compare generated text against reference answers using NLP metrics, directly supporting hallucination detection. Built-in evaluators also assess groundedness and relevance as part of the RAG quality evaluation category. An LLM-based evaluator can label each response with categorical labels such as factual vs. hallucinated, along with explanations and actionable feedback.

Policy and Safety Scoring

Content Safety in Microsoft Foundry’s control plane provide enterprise-grade harm detection with severity scoring from 0–7 and detailed reasoning for each flag. Built-in safety evaluators identify potential content and security risks in generated output, while the Violence evaluator specifically checks whether responses contain violent content. By default, Foundry automatically applies a baseline safety guardrail to all models and agents, covering hate and fairness, sexual and violent content, self-harm, protected text or code usage, and prompt injection attempts. Organizations can also create custom guardrails to tune sensitivity levels, selectively enable or disable specific risk categories, and define what actions to take annotating or blocking content.

4. Quantitative Health Metrics for AI Agents

Beyond qualitative evaluation, agents require quantitative metrics that extend traditional infrastructure monitoring. The following table summarizes key metrics grounded in current platform capabilities:

Trade-off note: Comprehensive per-request evaluation (running an LLM-as-judge on every response) adds latency and cost. Teams typically balance real-time lightweight safety checks on all responses with sampled deeper evaluations. This mirrors the two complementary evaluation paths: dataset-driven (curated test sets before deployment) and trace-driven (analyzing real model responses in production).

💡 See these metrics in action: Explore the https://learn.microsoft.com/azure/foundry/observability/how-to/how-to-monitor-agents-dashboard to view live metrics like latency, token usage, success rate, and safety scoring for your agents.

5. How Microsoft Foundry Enables Agent Observability

Microsoft Foundry is described as “the AI app and agent factory” – a unified, interoperable platform for building, optimizing, and governing AI apps and agents. Its observability capabilities span several integrated layers:

Built-in Evaluations: Foundry provides pre-built evaluators across multiple categories: general purpose, textual similarity, RAG quality, safety and security, and agent quality. AI-assisted evaluators like Task Adherence and Coherence use a GPT model deployment (e.g., gpt-4o or gpt-4o-mini) as the underlying judge. Teams can also build custom evaluators for domain-specific requirements. The evaluation service sends each test query to the agent, captures the response, and applies selected evaluators to score results automatically.

Try it yourself: Run an evaluation from the https://learn.microsoft.com/azure/foundry/how-to/evaluate-generative-ai-app (starting from the Evaluation page, Models page, Agents page, or Agent playground) or follow the https://learn.microsoft.com/azure/foundry/observability/how-to/evaluate-agent for step-by-step guidance on choosing evaluators and interpreting results.

Continuous Monitoring: Azure Monitor, in partnership with Foundry, unifies agent telemetry with infrastructure, application, network, and hardware signals — creating an end-to-end operational view. Specific capabilities include an Agent Overview Dashboard (available in Grafana and Azure) tracking success rate, grounding quality, safety violations, latency, and cost per outcome, and Foundry Low-Code Agent Monitoring where agents built through Foundry’s visual interface are automatically observable without additional instrumentation code. All evaluations, traces, and red-teaming results are published to Azure Monitor, where agent signals correlate with infrastructure KPIs.

Responsible AI and Governance Alignment: Every agent is assigned a Microsoft Entra Agent ID, enabling IT teams to apply Conditional Access, Identity Protection, and Identity Governance policies to agents just as they would to human users. Foundry enforces keyless authentication through Microsoft Entra ID, supports data encryption at rest and in transit (with Customer Managed Keys for greater control), and offers network isolation via Managed VNet or Bring Your Own VNet. The AI Red Teaming Agent enables proactive adversarial testing to uncover jailbreaks, prompt injection attacks, and other security weaknesses before deployment.

6. Applying Observability Across the AI Lifecycle

Historically, evaluation and monitoring were treated as separate phases: data scientists tested models offline, and engineers observed them after deployment. With LLMs and non-determinism, this divide no longer works. Observability must be woven into every lifecycle stage:

Design-Time Evaluation: During development, establish quality baselines by creating test datasets and defining acceptance thresholds. Foundry enables teams to set criteria such as requiring an 85% task adherence passing rate before releasing an agent to users. Run evaluators iteratively during prompt engineering to catch regressions early.

Pre-Production Validation: Agent CI/CD must handle model versioning, evaluation against behavioral benchmarks, and non-deterministic behavior challenges absent in traditional software pipelines. Unlike conventional unit tests that verify exact outputs, agent evaluation must assess variable responses holistically across dimensions: task completion rate, tool usage accuracy, response quality, latency, and cost. Adversarial testing intentionally testing edge cases such as ambiguous requests, tool failures, and conflicting information  is essential because production agents will encounter these scenarios. Foundry’s AI Red Teaming Agent supports this directly.

💡 Learn more: Explore the https://learn.microsoft.com/azure/foundry/how-to/develop/run-ai-red-teaming-cloud to see how to proactively test your agent against potential attacks and edge cases before production.

Runtime Monitoring: Deploy using progressive delivery patterns such as canary deployment (routing 5–10% of traffic to the new version) with automatic rollback if any metric degrades beyond thresholds. In production, the continuous feedback system means the same evaluators that power offline testing also monitors live production traffic, and data moves seamlessly from trace logs to evaluation results to dashboards. Track every input, output, API call, token usage, and decision point from day one — patterns showing which metrics predict failures only emerge after accumulating production data.

Continuous Improvement: Export low-scoring interactions for analysis and use them to refine prompts, update knowledge bases, or fine-tune models via Azure Machine Learning. The goal is a closed loop: monitoring reveals issues → issues trigger investigation and updates → updates are re-evaluated in staging → improvements are redeployed. This transforms Responsible AI from policy into practice.

7. Modernization Perspective: Leveraging LLMs for Observability

LLMs as Evaluators and Explainers

LLM judges produce scores, rankings, categorical labels (e.g., factual vs. hallucinated), explanations, and actionable feedback — enabling iterative refinement of AI applications. This scalable, consistent approach reduces dependency on human annotations by providing interpretable explanations. LLM evaluators use three main input types: pointwise (evaluating one output at a time), pairwise (comparing two outputs), and listwise (ranking multiple outputs). Assessment criteria span linguistic quality (fluency, coherence), content accuracy (fact-checking, logical consistency), task-specific metrics (completeness, informativeness), and user experience.

Benefits and Limitations

Mitigation strategies recommended by the research community include: regularly auditing for bias and fairness, incorporating domain experts in the evaluation process, standardizing prompt designs to reduce variability, and combining human oversight with automated evaluation systems. The key takeaway is that human-AI collaboration enhances reliability and enables iterative improvements. Architects should design their observability systems to use LLM-based evaluation as a force multiplier for human reviewers, not as a replacement.

8. Conclusion: Building Trustworthy, Observable AI Systems at Scale

Observability for agentic AI systems requires a fundamental expansion of what is monitored and how. Microsoft’s approach with Foundry and Azure Monitor demonstrates an integrated model: agents are instrumented to capture every reasoning step, LLM-powered evaluators assess quality and safety continuously, and all signals flow into the same enterprise monitoring infrastructure used for traditional applications. By building on OpenTelemetry open standards including Microsoft’s contributions to the OpenTelemetry agent specification for multi-agent orchestration traces and LLM reasoning context — organizations avoid vendor lock-in while gaining consistent visibility across multi-cloud environments.

The practical implication for architects: design for observability from day one. Instrument reasoning traces, define quality and safety evaluators early, automate evaluation in CI/CD, and establish continuous monitoring with clear escalation thresholds. 

As an immediate next step, teams can enable the https://learn.microsoft.com/azure/foundry/observability/how-to/how-to-monitor-agents-dashboard to begin tracking token usage, latency, success rate, evaluation scores, and red teaming results for their first agent. For teams still in the planning phase, assembling an observability checklist covering trace instrumentation, evaluator selection, safety baseline configuration, cost alerting, and governance controls ensures no critical signal is left uncaptured when agents reach production.

The organizations that bring the same rigor of testing, cost governance, compliance, and continuous improvement to their generative AI agents as they do to traditional applications will be the ones to ultimately earn and maintain trust in their AI systems at enterprise scale.

9. Get Started with Agent Observability

Ready to implement these observability patterns? Choose your path:

🚀 Quick Start

• Deploy an observable agent: Use the step-by-step https://learn.microsoft.com/azure/foundry/agents/quickstarts/quickstart-hosted-agent with Microsoft Foundry — it walks you through setting up a sample agent with the Azure Developer CLI and seeing monitoring in action. 


• Explore the Agent Monitoring Dashboard: Open the https://learn.microsoft.com/azure/foundry/observability/how-to/how-to-monitor-agents-dashboard for your Foundry project to view live metrics, traces, and safety checks as your agent handles requests.

📚 Go Deeper

• Master evaluations: Read the Microsoft Foundry https://learn.microsoft.com/azure/foundry/concepts/built-in-evaluators to understand the full catalog of quality, safety, and agent evaluators — and learn how to https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/custom-evaluators for domain-specific criteria. 


• Practice red teaming: Follow the https://learn.microsoft.com/azure/foundry/how-to/develop/run-ai-red-teaming-cloud to simulate adversarial scenarios and strengthen your agent’s defenses before and after deployment. 

🤝 Join the Community

• Foundry forums: Ask questions and share insights in the https://github.com/microsoft-foundry/discussions.


• GitHub samples: Explore the https://github.com/microsoft-foundry/foundry-samples for example agents, evaluations, and monitoring setups.


• Community chat: Connect with fellow developers on the https://discord.com/invite/microsoftfoundry to discuss best practices and get the latest tips.

📧 Stay Updated

• Azure AI Newsletter: Subscribe to the https://info.microsoft.com/ai-newsletter.html for monthly updates on new features, case studies, and best practices in Azure AI — including Microsoft Foundry and observability innovations.