Modern Ingress for AKS: Introducing Application Gateway for Containers (AGC)

Teamcenter on Oracle AI Database@Azure: Architecture, Validation, and Results

July 7, 2026

Streamlining business processes with Microsoft 365 – ShareCloud – SharePoint Partner Showcase

July 7, 2026

Published by azurefeeds on July 7, 2026

Why IT Pros Should Care

The NGINX Ingress Controller retirement is pushing a lot of teams to pick a new front door for their AKS clusters. AGC is Microsoft’s managed, Azure-native answer. It is Layer 7, it speaks both the Kubernetes Ingress API and the newer Gateway API, and it runs outside the cluster so you are not paying with pod CPU to terminate TLS.

Here is the short list of why this session matters:

The internet runs on load balancers, and the load balancer story has been mostly stable for decades. That is changing fast.

AI inference traffic does not look like web traffic. Long-lived, GPU-bound, expensive. Traditional Layer 7 rules cannot keep up.

AGC now ships AI gateway capabilities (in preview) that route by model, route by load, and stop slamming a busy pod when others are idle.

Measured numbers from the demo: zero timeouts versus 15 failed requests, and a P99 time-to-first-token of about 1,000 ms versus 16,000 ms on a vanilla HTTPRoute.

If your AKS clusters are starting to host self-hosted models alongside regular services, you need to know this exists.

What is Application Gateway for Containers

AGC is the evolution of the older Application Gateway Ingress Controller (AGIC). The architecture is the part that matters:

Application Gateway for Containers resource: the managed Azure resource that actually handles traffic, TLS, WAF, and routing.

Frontends: listeners and IP addresses that accept incoming traffic.

Associations: the link between the AGC resource and your AKS VNet/subnet.

ALB Controller: a small controller that runs inside your AKS cluster, watches your Gateway API and Ingress resources, and reconciles the AGC configuration in near real time.

Two deployment models. You can let the ALB Controller manage the lifecycle of the AGC resource for you, or you can go Bring Your Own (BYO) and manage it with Bicep, Terraform, ARM, or the CLI. I like BYO for production because it gives the platform team a clean separation between Kubernetes intent and the Azure resource lifecycle.

Feature-wise it covers what you would expect from a modern Layer 7: automatic retries, autoscaling, zone resiliency, weighted/split traffic for blue-green, header and URL rewrites, WAF integration, mTLS (front, back, or end-to-end), HTTP/2, WebSocket, gRPC, and server-sent events. Routing by hostname, path, header, query string, or HTTP method.

How AGC works with AKS

Jack opened with a tour of the OSI model that I genuinely enjoyed. The TL;DR:

Layer 4 (network load balancer): stateless, fast, simple. Source IP, destination IP, send. This still powers most of the internet.

Layer 7: TLS, session affinity, WAF, mutual auth, intelligent routing based on the contents of the request. This is where modern web applications live.

Kubernetes ingress: the Layer 7 load balancer is no longer a hardware appliance you rack and stack. Pods come and go constantly. You need a controller that translates Kubernetes intent into load balancer config in near real time.

Gateway API: the CNCF’s vendor-neutral way to express that intent. AGC is a first-class Gateway API implementation, so the YAML you write is portable across clouds and on-prem.

In Gateway API land you work with three primary resources:

A Gateway (the front door, including listeners and hostnames).

An HTTPRoute that says “for requests matching this host/path/header, send them to this backend.”

A backend, traditionally a Kubernetes Service.

That is the picture for normal web apps. AI workloads break it.

Real-world value

Here is the honest tradeoff Jack laid out. Traditional Layer 7 load balancers were tuned for short-lived, low-latency requests. They distribute traffic with simple algorithms (round robin, least connections) because they assume backends are interchangeable.

AI inference flips every one of those assumptions:

Requests are long-lived. A model reasoning over a big prompt can take many seconds.

Backends are not interchangeable. Pod A might be running an image model, Pod B a coding model.

Compute is expensive. GPUs are not cheap, and a hot pod with a full KV cache cannot just accept more work.

A traditional load balancer will happily pile new requests on a pod that is already saturated, while idle pods sit waiting.

The fix is the CNCF Gateway API Inference Extension. It introduces four primitives, and AGC implements them so you do not have to bolt the pieces together yourself:

Body-Based Router (BBR): reads the prompt body to pull out the requested model. In AGC, this runs inside the load balancer itself. Nothing extra for you to deploy.

Endpoint Picker (EPP): runs in your cluster, scrapes metrics from your model pods (things like KV cache utilization), and tells the load balancer which pod is actually a good target right now.

InferencePool: a new Kubernetes resource that groups pods serving the same model along with the EPP and target port.

InferenceObjective: lets you express priority when multiple requests are competing for the same pool.

The neat part is that InferencePool plugs in as a backendRef on an HTTPRoute, right next to traditional Service backends. You can mix AI and standard workloads behind the same gateway. No new gateway, no new product, just an additional backend type.

Jack ran two demos that I want to call out:

Model-aware routing: a script fires curl requests with different prompts. AGC’s body-based router pulls the requested model out of the JSON body, sets a header, the HTTPRoute matches the header, and traffic lands on the right InferencePool. The vanilla HTTPRoute side cannot do this and a lot of requests fail.

Load-aware routing: three pods, one already busy. Fire 50 requests at the AI-enabled route and 50 at the traditional route. The endpoint picker steers traffic to the idle pods. P99 time-to-first-token: about 1,000 ms with AGC’s AI gateway, about 16,000 ms without. Timeouts: zero with AGC, 15 without.

That is a real number, on the same backend, in the same cluster. The only thing that changed is the load balancer knowing what is happening behind it.

Getting Started

If you want to try this, here is the path I would take:

Stand up an AKS cluster with Azure CNI, OIDC issuer, and Workload Identity enabled. AGC needs these.

Register the providers: Microsoft.ContainerService, Microsoft.Network, Microsoft.NetworkFunction, Microsoft.ServiceNetworking.

Create a user-assigned managed identity for the ALB Controller and grant it the AGC Configuration Manager role.

Install the ALB Controller via Helm or as an AKS add-on, then create your Gateway resource against the azure-alb-external GatewayClass.

Start with a plain HTTPRoute against a regular Kubernetes Service. Get that green first.

When you are ready for inference, deploy your model, drop in an InferencePool and an Endpoint Picker, and point your HTTPRoute at the pool.

Pierre’s tip: the AI gateway capabilities are in preview. The team explicitly asked for feedback from anyone running self-hosted models in AKS, especially around model mix, request patterns, and what you are doing today to keep hot pods from getting buried.