AzureFeeds

By Shantanu Patankar, and Azin Heidarshenas

As part of Azure’s MLPerf Training v6.0 submission, we scaled Llama 3.1 405B (NVFP4) pretraining to 8,192 GB200 GPUs across 128 racks on Azure’s Fairwater infrastructure, converging in 7.07 minutes (MLPerf Training Blog). Llama 3.1, with 405 billion active parameters, has the highest active parameter count among all models in the MLPerf suite. 

Getting a model of this scale to train efficiently requires a cascade of tightly coupled choices around parallelism, batch sizing, and network placement. This post shares what we learned from bringing up and scaling Llama 3.1 405b on Fairwater at frontier scale. Three insights emerged: 

1. Even at 8K+ GPU scale, Llama 3.1 405B training remains compute-bound, with Feed forward Network (FFN) kernels dominating execution


2. Topology-aware mapping is the key enabler of 8000+ GPU scale


3. Scaling efficiency is governed by convergence dynamics, not system or network limits 

The rest of this post explains how these effects appear in practice. 

Where Does the Time Go Inside One Iteration, and What Are the Bottlenecks? 

Llama 3.1 405B is a dense transformer with 126 layers, each consisting of a self-attention block (QKV projections and attention) followed by a feed-forward network (FFN). To understand what limits performance at scale, we profiled steady-state iterations and broke down where time is spent, as shown in Figure 1. A single training step (Global Batch Size=1024) takes approximately 1.27s. 

Compute:  

Roughly 57% of the training iteration wall time (~733 ms) is spent in compute. GEMM kernels dominate, accounting for 74% of that compute time. What is striking is how concentrated the compute is. The FFN up-projection and down-projection GEMMs alone consume ~54% of total compute (~397 ms). Everything else, including QKV projections, attention, softmax, normalization, etc., shares the remaining 46%. the FFN clearly dominates and is the primary driver of compute time. 

Communication: 

The system operates across two communication domains: high-bandwidth NVIDIA NVLink (NVL) within a rack (scale-up, 72 GPUs) and Azure’s Multipath Reliable Connection (MRC) network across racks (scale-out). The total communication work is 853 GPU-ms. Nearly 42% of this overlaps with compute or other communication (with MRC overlapping NVL), leaving only ~500 ms on the critical path. This is the portion that directly impacts step time, as shown in Figure 1. Of this ~500 ms critical-path communication, NVLink accounts for nearly all of it, while MRC contributes just 1.6%.

Intra-rack NVLink communication on the critical path accounts for 37.8% (~479 ms). This includes Tensor Parallelism (TP), Pipeline Parallelism (PP), and Context Parallelism (CP) exchanges on the 1.8 TB/s intra-rack NVLink fabric, as well as pipeline bubbles, where stages idle between micro-batches. Although substantial, in a weak scaling regime, where each rack handles a fixed shard of the workload, NVLink communication per rack remains constant as we scale.

MRC communication covers gradient synchronization across all DP ranks. Although it represents a substantial amount of communication work, it overlaps heavily with both NVLink communication and compute. As a result, only 1.6% (~20 ms) remains on the critical path. The cross-rack MRC network is nearly invisible in the step profile.

The workload is compute-bound even at 8192 GPU scale, with FFN as the center of gravity. Any improvement to FFN kernel efficiency (precision, tiling, scheduling) translates almost directly into faster training steps.  

Why Is Topology-Aware Mapping the Key Enabler of Scale? 

In the first section, we showed that cross-rack MRC communication accounts for just 1.6% of step time. This result comes from how parallelism is mapped to the cluster’s two-tier network.

Training Llama 3.1 405B distributes work across four dimensions. Tensor Parallelism (TP), Pipeline Parallelism (PP), and Context Parallelism (CP) split the model and sequence across GPUs, communicating frequently throughout each step. Data Parallelism (DP) replicates the model and synchronizes gradients once per step. The key difference: TP, PP, and CP are latency-sensitive. DP communication can be overlapped with compute, along with other NVLink-based communication. The design principle follows directly: keep latency-sensitive dimensions on the fastest links, and push traffic that can be overlapped with compute to the scale-out fabric

On Azure’s Fairwater cluster, each rack provides a 72-GPU NVLink domain at 1.8 TB/s. We use 64 GPUs per rack and constrain TP × PP × CP = 64, so one complete model-parallel shard fits within a single rack. Scaling to more racks adds DP replicas, with gradient synchronization flowing across the 100 GB/s Multipath Reliable Connection (MRC) network, the traffic that hides almost entirely behind compute. 

We swept TP, PP, and CP at fixed DP = 8 and GBS = 1152 to isolate the effect of communication efficiency on throughput. TP4–PP8–CP2 achieves the highest TFLOPS/GPU among the feasible configurations, reaching an Model FLOPs Utilization (MFU) of 26.45% as shown in Figure 2. 

What Limits Scaling Efficiency at 8K+ GPU Scale? 

We scaled Llama 3.1 405B training from 40 to 128 racks, adjusting MiniBS (per-DP-shard batch size, i.e., Global Batch Size ÷ DP) and Global Batch Size (GBS) at each scale point to balance GPU utilization against convergence efficiency. Training time drops from 18.79 minutes to 7.07 minutes, representing 83% scaling efficiency relative to the 40-rack baseline as shown in the table below.

Notice that MiniBS shrinks as we scale. MiniBS determines how much work each GPU processes per step and is the primary driver of step time. As we scale by increasing DP (one replica per rack), the global batch size grows proportionally unless MiniBS is reduced. 

Higher MiniBS improves per-step GPU utilization, but at scale it inflates GBS, and beyond a critical batch size the model requires quadratically more samples to converge. So we trade per-step throughput for faster end-to-end convergence, reducing MiniBS from 144 at 8 racks to 8 at 128 racks. 

MiniBS cannot be reduced below 8 in our setup because PP = 8 requires at least one microbatch per pipeline stage to keep the pipeline fully utilized. Since MiniBS cannot drop further, adding more racks only increases GBS, pushing deeper into the convergence penalty. At large scale, the optimal configuration minimizes time-to-train, not per-step throughput.  The 83% scaling efficiency gap is not a network or system bottleneck. It is inherent to the benchmark’s convergence dynamics. 

Conclusion 

MLPerf Training v6.0 provides a practical lens into how frontier-scale LLM systems behave under real constraints. Azure’s Llama 3.1 405B submission, which achieved a leading result at 8K+ GPU scale, highlights how performance at this level is shaped not just by infrastructure, but by the interaction of compute, topology, and convergence. Scaling efficiently at this level requires aligning these factors with the workload, not just adding more compute.

• View Azure’s MLPerf Training v6.0 results.


• Read Azure’s MLPerf v6.0 announcement blog.
  
  

Acknowledgement

This work was made possible through strong collaboration across multiple teams. We thank Prabhat Ram for his guidance throughout this effort, along with Mark Gitau, Amirreza Rastegari, Ojasvi Bhalerao, Sai Kovouri, Adam Hough, Bhupender Thakur, Nandini Ramanathan, Manasa Govindu, Sanian Gaffar, Ekrem Aksoy, Matthew Kappel, John Rankin, Girish Bhatia, Sreevatsa Anantharamu, Scott Moe, Yang Wang, and Jithin Jose — as well as many others across Azure and NVIDIA who supported this effort.