AzureFeeds

This post will take you one step further from the hello-world configuration that I described in the first post by adding TLS end-to-end, a scenario described in the public docs here. This blog is part of a series:

• 1. Introduction to AGC architecture and components
• 2. Troubleshooting
• 3. Resource model expansion: end-to-end TLS (this post)

TL;DR

If you don’t have much time the diagram below, which is an extended version of the simplified diagram in the first post of the series, summarizes the main content:

Without any more delays, let’s dive in into the most salient features of the extension to the object model presented in part #1 of the series.

Where are the listeners?

Since most objects in the model of the traditional Azure Application Gateway have their counterpart in Kubernetes Gateway API, I had also expected to see a Kubernetes resource to model the listeners. However, that is not the case: instead they are a part of the gateway resource (note that I use variables inside of the YAML, please refer to my Azure CLI script to more details on how those are initialized):

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: $agc_name
  annotations:
    alb.networking.azure.io/alb-namespace: $alb_ns_name
    alb.networking.azure.io/alb-id: $alb_id
spec:
  gatewayClassName: azure-alb-external
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
  - name: https
    port: 443
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind : Secret
        group: ""
        name: $secret_name

As you can see, there are two listeners in the gateway. As we saw in post #1, you don’t configure the FQDNs or URLs in the listener, but in the HTTP route. However, most of the HTTP route examples in the docs don’t have any mention about the listener, how can you force a route to be attached to one listener but not to other? The answer is the sectionName attribute. If you don’t specify any sectionName your HTTP route will be attached to all listeners. Here you have my HTTP route with the TLS backends that will only be attached to the https listener:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: tlsyadaapi
  namespace: $tls_app_ns
spec:
  parentRefs:
  - name: $agc_name
    namespace: default
    sectionName: https
  hostnames:
  - "$tls_app_fqdn"
  rules:
  - backendRefs:
    - name: yadatls443
      port: 443

Certificates

Certificates are stored as secrets, which as you saw in the previous section are referenced from the frontend as part as the gateway’s spec:

In my tests I created the secret from local files containing the public and private key for my digital certificate, which I downloaded from DigiCert. You would have those files too if you are using self-signed certs:

kubectl create secret tls $secret_name --cert=$pem_file --key=$key_file

Of course, you can (and you should) use more sophisticated like cert-manager (see the Microsoft docs on how to do this here) or the Azure Key Vault provider for secrets store CSI driver (that is quite a mouthful, that is one of the projects that in my opinion needs an urgent renaming).

Backend settings

The concept in the gateway API is very similar to the model of Azure Application Gateway. Again, like HTTP routes and health check policies, you need to place the backend TLS policy resource in the same namespace as the service that it targets:

Here the backend TLS policy I am using for my sample app. You can define multiple things like the content you want in the Server Name Indication (SNI) TLS header or the backend port:

apiVersion: alb.networking.azure.io/v1
kind: BackendTLSPolicy
metadata:
  name: yadatlspolicy
  namespace: $tls_app_ns
spec:
  targetRef:
    group: ""
    kind: Service
    name: yadatls443
    namespace: $tls_app_ns
  default:
    sni: $tls_app_fqdn
    ports:
    - port: 443

You can see more of the attributes you can define in the backend TLS policy in the reference page. For example, this is where you would refer to a secret containing the Certificate Authority (CA) that signs the backend’s certificate, if it is not one of the well-known CAs that AGC trusts per default.

With the backend TLS policy and the health check policy (which I need, because otherwise AGC would poll for the root URL, to which my app is not responding), I see now in the AGC metrics in the Azure portal that the backend yadatls443 is healthy:

Frontend TLS policy

Similarly to the backend TLS policy you can define frontend TLS policies, for example to specify that mTLS should be used (see the frontend mTLS example in the docs):

This doesn’t seem to be a resource part of the gateway API, so you will not find information about this resource in the Gateway API docs. In my test I am not using mTLS, so I don’t need frontend TLS policy.

My workload

I am still using the Microsoft YADA API container image for my test app, but to test end-to-end TLS I deploy nginx as a sidecar, where I install the same certificate as in the frontend. For more details on how to that you can have a look at my test script.

This is important because when testing the application I will see in the X-Forwarded-For header the IP addresses of each of the reverse proxies involved: the AGC and the nginx side car:

❯ curl --resolve "${tls_app_fqdn}:443:${ip_address}" "https://${tls_app_fqdn}/api/ip"
{
  "host": "yada.cloudtrooper.net",
  "my_default_gateway": "169.254.1.1",
  "my_dns_servers": "['10.0.0.10']",
  "my_private_ip": "10.13.76.32",
  "my_public_ip": "135.224.179.67",
  "path_accessed": "yada.cloudtrooper.net/api/ip",
  "sql_server_fqdn": "None",
  "sql_server_ip": "False",
  "x-forwarded-for": "['93.104.172.225, 10.13.100.6']",
  "your_address": "127.0.0.1",
  "your_browser": "None",
  "your_platform": "None"
}

Finally, I can check the AGC logs (you need to have configured diagnostics settings in Azure) that all is working as expected. As you can see below, I have two listeners, one answering for the unencrypted version of my application and the other for TLS:

❯ query="AGCAccessLogs | where TimeGenerated > ago(5m) | project TimeGenerated, ClientIp, HostName, RequestUri, FrontendName, FrontendPort, BackendHost, BackendIp, HttpStatusCode"
❯ az monitor log-analytics query -w $logws_customerid --analytics-query $query -o table
BackendHost      BackendIp    ClientIp              FrontendName    FrontendPort    HostName                             HttpStatusCode    RequestUri        TableName      TimeGenerated
---------------  -----------  --------------------  --------------  --------------  -----------------------------------  ----------------  ----------------  -------------  ------------------------
10.13.76.32:443  10.13.76.32  93.104.172.225:55094  test-frontend   443             yada.cloudtrooper.net                200               /api/ip           PrimaryResult  2025-03-08T08:47:58.976Z
10.13.76.7:8080  10.13.76.7   93.104.172.225:55080  test-frontend   80              dbfehra5cdcqdabc.fz38.alb.azure.com  200               /api/healthcheck  PrimaryResult  2025-03-08T08:32:11.452Z

Of course, in your production app you wouldn’t want to have a listener that allows unencrypted access to your app, but a redirect listener that forces users to use TLS. You can see more information about redirects in the Gateway API here, or check the TLS example in the AGC docs.

Conclusion

You have seen that by using TLS you will need additional resources to the ones we described in our hello-world post, such as backend TLS policies, secrets, a tighter control of which HTTP routes attach to which listeners, and potentially frontend TLS policies.

Have I forgotten any advanced gateway API resource that I should have covered? Let me know in the comments!