AzureFeeds

Building on our initial announcement for Deploying Open Source Software on Azure, the AKS and the Customer Experience team are excited to announce our expanded library of technical best practice deployment guides for stateful and AI workloads on AKS. 

These guides are designed to help you accelerate the integration of some of the most critical and heavily adopted open source projects onto Azure, utilizing best practices and optimizations for AKS. For your convenience, you can jump to our collection of Stateful and AI guides at the bottom.  

To receive updates and read about the other improvements and updates please follow us at the AKS Engineering Blog. In the AKS Engineering Blog, we discuss our updated Postgres guidance with additional storage considerations for data resiliency, performance, or costs using Azure Container Storage. We also highlight our Terraform additions for the Mongo DB and Valkey guides, and the newly developed Azure Verified Module for deploying a production-grade AKS Cluster.

Introducing New Guidance

Below is a brief overview of each new guide available today. In addition to basic installation steps, the guides detail best practice recommendations on networking, monitoring, and security. The guidance includes initial setup to advanced AKS cluster and node configurations, ensuring applications are successfully deployed with best practices enabled. 

Deploy Kafka on AKS with Strimzi for Distributed Streaming

Apache Kafka is an open source distributed event streaming platform designed to handle high-volume, high-throughput, and real-time streaming data. It is widely used by thousands of companies for mission-critical applications but managing and scaling Kafka clusters on Kubernetes can be challenging. Strimzi simplifies the deployment and management of Kafka on Kubernetes by providing a set of Kubernetes Operators and container images that automate complex Kafka operational tasks. 

The Kafka on Azure Kubernetes Service (AKS) guide covers essential storage and compute considerations, ensuring you Kafka deployment meets your needs. Additionally, we provide guidance for tuning the Java Virtual Machine (JVM), which is critical for optimal Kafka broker and controller performance. 

Deploy Airflow on AKS for Workflow Orchestration 

Apache Airflow is a widely adopted open source workflow orchestrator often used for ETL processes, data pipeline management, and task automation. Despite its interoperability and importance, deploying Airflow on Kubernetes is very complex. 

As best practice, we recommend installing Airflow using the community-maintained Helm Chart. Deploying Apache Airflow on Azure Kubernetes Service (AKS), provides step-by-step instructions for installing the Helm Chart, configuring the Azure Container Registry and setting up user-assigned managed identities. Additionally, we provide recommendations on adjusting default configurations to optimize Airflow on AKS.  

Deploy Ray on AKS for Distributed Computing 

Ray is an open source framework for distributed computing, allowing both Python applications and common machine learning workloads to be parallelized and distributed efficiently.   

The Ray on Azure Kubernetes Service (AKS) guide details how to configure the underlying infrastructure and deploy KubeRay, the Ray Kubernetes Operator. KubeRay is the best way to deploy and managed Ray on AKS as it provides a Kubernetes-native way to manage Ray clusters. Additionally, this guide includes two illustrative samples: one for distributed training and another for fine-tuning. The distributed training example demonstrates a PyTorch model trained on Fashion MNIST using Ray Train on AKS. The fine-tuning example showcases how to tune the GPT-2 large model using KubeRay in combination with Azure Blob Storage and BlobFuse. 

Deploy open source technologies on Azure 

• Apache Airflow – Create the infrastructure for deploying Apache Airflow on Azure Kubernetes Service (AKS) 


• Apache Kafka – Prepare the infrastructure for deploying Kafka on Azure Kubernetes Service (AKS)  


• Ray – Deploy a Ray cluster on Azure Kubernetes Service (AKS) 


• Valkey – Create the infrastructure for running a Valkey cluster on Azure Kubernetes Service (AKS) 


• Mongo DB – Create the infrastructure for running a MongoDB cluster on Azure Kubernetes Service (AKS) 


• Postgres – Create infrastructure for deploying a highly available PostgreSQL database on AKS 


• Kubernetes AI Toolchain Operator (KAITO) – Deploy KAITO on AKS using Terraform