AzureFeeds

Abhishek Tiwari, Vice President of Engineering, Azure Networking
Amit Srivastava, Partner Director of PM, Azure Networking
Varun Chawla, Partner Director of Engineering, Azure Networking

Azure Front Door serves hundreds of thousands of tenants from hundreds of edge locations, densely sharing the globally distributed edge fleet. That density is exactly what allows us to deliver global scale, performance, and cost efficiency. It also means that, without strong isolation, a single tenant’s incompatible configuration or anomalous traffic can, in the worst case, affect many other tenants. The October 2025 incidents reinforced how important it is to contain this class of risk. Our goal for this tenant isolation is simple to state and hard to achieve: no single tenant’s configuration or traffic should be able to impact any other tenant.

In Part 1 of our three part mini blog series, we outlined our four‑pillar strategy for improving the resiliency of Azure Front Door: configuration resiliency, data plane resiliency, tenant isolation, and accelerated Recovery Time Objective (RTO). Part 1 detailed how we would make configuration propagation safer and how the data plane keeps serving from a ‘last‑known‑good’ (LKG) configuration, even if an incompatible configuration change is propagated to the data plane. Part 2 turned to recovery, showing how we bring the system back to full operation in an accelerated, predictable, and in a bounded timeframe. In this final part, we turn to the tenant isolation pillar that ensures that any single tenant configuration or traffic issues are limited in scope to that tenant alone and do not impact other tenants. We will also show how Azure Front Door achieves single-tenant containment through configuration isolation, lazy loading, and a micro-cellular layered ingress-sharding architecture.

Repair status: all outstanding items now complete

Before we dive into tenant isolation, here is a final update on the overall repair items from the two October 2025 incidents (you can review the details in our Azure Incident Retrospective sessions for the October 9th and October 29th incidents). We are pleased to report that all outstanding work across every pillar is now complete and fully deployed in production – including the tenant isolation work described in this post. With these safeguards in place, we have also returned configuration propagation latency to pre‑incident levels while keeping platform stability our top priority. In the table below, “Completed” means broadly deployed in production.

Why isolation at edge scale is deceptively hard

Traditional isolation techniques such as dedicating separate hardware to each tenant or running every tenant inside its own virtual machine are impractical at the edge. Edge sites are constrained on space, power, and capacity. The entire premise of a modern, multi-tenant application delivery platform is that any tenant can be served from any site closest to the user. We cannot simply partition hundreds of thousands of tenants onto dedicated machines without giving up either proximity, scale, or the efficiency that make the edge fast and cost efficient. Isolation therefore, must be achieved in software, inside a multi-tenant fleet.

What we already do today

Azure Front Door already includes several layers of tenant isolation and partitioning. However, the incidents in October clearly highlighted that these techniques were not enough. Prior to the incidents, our protection mechanisms included:

• Infrastructure partitioning. Edge sites were organized into physically isolated primary and fallback traffic rings.


• Noisy‑neighbor protection. Fair‑share resource allocation, rate limiting, and anomaly-based load protection kept any single tenant from monopolizing shared resources such as CPU, memory, or network bandwidth on an Azure Front Door server.


• Circuit breakers. Circuit breakers shed costly work first and can disable a risky per‑tenant feature before it exhausts shared resources on a server.


• Real‑time crash protection. A crash‑analysis system correlates crash signatures across machines and can pinpoint and block crash patterns caused by tenant IPs or traffic patterns.

While these protections are valuable, many of them are reactive and proved insufficient during the October incidents. The next generation of isolation makes single‑tenant containment a fundamental part of the platform which governs how configurations are loaded, and how traffic is served.

Configuration isolation: loading only what is needed

Part 2 introduced ‘lazy loading’ as a recovery optimization technique. It is also an important configuration‑isolation mechanism. Historically, every worker on every edge server had to be ready to serve any tenant, which meant each worker loaded a large set of tenant configurations. A single incompatible configuration could therefore ripple across many workers.

With lazy loading, a worker loads a tenant’s configuration and its TLS certificates only when it actually receives traffic for that tenant. The practical consequence for isolation is powerful: a faulty configuration can only affect the workers that have loaded that specific tenant, never the entire server or fleet. Combined with per‑tenant validation on load, and the Food Taster safeguard from Part 1 (a sacrificial process that pretests every configuration change in isolation), configuration problems are caught early and contained to the smallest possible footprint.

Figure 1: With lazy loading, an incompatible configuration is contained to the workers serving that tenant, instead of poisoning the whole server.

Tenant isolation: a micro-cellular, layered ingress sharding architecture

Configuration isolation limits the blast radius of an incompatible configuration. Traffic isolation addresses the other half of the problem: a tenant whose anomalous traffic incident, like a sudden surge, a pathological request pattern, or malicious activity, could degrade a shared worker. Our approach is a micro‑cellular architecture that combines multiple concepts working together.

• Worker‑process isolation. Each edge server already runs many independent worker processes. Instead of letting every worker serve every tenant, we assign tenants to specific groups of workers. Those worker group (shards) become the unit of isolation: if a tenant destabilizes its shard, the impact is contained to that shard’s workers while the rest of the server keeps serving normally.


• Ingress sharding. Rather than a handful of fixed shards, we compose shards from overlapping subsets of a server’s workers. Even a modest number of workers can be combined into an enormous number of distinct, overlapping shards – giving us a very large number of fine‑grained fault domains without dedicating hardware to every tenant.

Figure 2: Tenants are randomly assigned to different shards on each server (layer). Even when a good tenant shares the noisy tenant’s shard on one layer, routing steers its traffic to healthy shards on the others.

• Multi‑layer ingress sharding. This is where ‘layered’ comes in. Each edge server is treated as an independent layer, and each tenant is assigned to a different, randomly chosen shard on every server. Because assignments are independent from one server to the next, two tenants that happen to share a shard on one server are extremely unlikely to share a shard again on another server. The chance of any good tenant repeatedly colliding with a noisy tenant across many servers becomes vanishingly small.


• Intelligent ingress routing. Tying it together is a routing layer that terminates each incoming connection, identifies the tenant, and steers the request to that tenant’s assigned, healthy shard. If a shard is unhealthy or saturated, traffic is directed to the tenant’s healthy shards on other layers.

Figure 3: Intelligent Ingress Routing

The combined effect is that when a noisy tenant overwhelms or crashes its shard on one server, only that shard is affected. Because every other tenant is spread across a different, randomized set of shards, they continue to find healthy paths, and the routing layer moves their traffic accordingly. A worst-case availability problem is downgraded to, at most, a small and redistributable capacity problem, that the routing layer smooths over.

An in-depth technical analysis of layered ingress sharding is available here for reference.

Shrinking the blast radius

Taken together, these mechanisms fundamentally change the shape of failure. In a uniform, fully shared fleet, an incompatible tenant can, in the worst case, affect a large share of the tenants on a machine, in an edge location, or beyond. With configuration isolation and layered ingress sharding, the same failure is confined to a subset of workers serving the offending tenant. Our target for this tenant isolation pillar is effectively single‑tenant containment: a configuration or traffic anomaly caused by one tenant should never cause issues to any other tenants.

Figure 4: From a shared fleet where a single tenant can affect many, to micro‑cellular shards that confine impact to the offending tenant.

Validating isolation in practice

As with our recovery work, we don’t simply design these boundaries and assume they hold, we test them. Through deliberate fault‑injections, we have pushed noisy and faulty tenants into the system and confirmed that impact stayed contained to the offending shard, that healthy tenants kept serving, and that the routing layer steered around unhealthy shards as intended. This turns isolation from a design claim into a well-drilled, and repeatable outcome.

Closing

This post concludes our three-part mini blog series on Azure Front Door resiliency. We have shared how we are making configuration propagation safer (Part 1), recovering faster when failures do occur (Part 2), and containing the blast radius stemming from any single tenant through configuration isolation and a micro‑cellular, layered‑sharding architecture (Part 3).

Resiliency, however, is not a project with an end date. It is an ongoing commitment. While this series concludes the blog series of our response to the October 2025 incidents, our investments in Azure Front Door’s resiliency, isolation, and recovery will continue. As we make further improvements, we will keep sharing them with you.

We deeply value our customers’ trust in Azure Front Door. We remain committed to exceeding expectations for security, reliability, and transparency.