I recently reviewed and reorganized my notes on distributed LLM training, drawing from Part 3 of Google's JAX Scaling Book[1], 猛猿's four illustrated walkthroughs on DP/DDP[8], DeepSpeed ZeRO[9], Megatron-LM tensor parallelism[10], and Megatron mixed-precision internals[11], alongside the foundational papers[2][5][6][7]. The sections walk straight through HBM memory and mixed precision, device meshes and the four collectives, DDP/ZeRO, TP, PP, and MoE before sizing a 64-GPU cluster.

Memory & Mixed Precision

Where Memory Goes

Following ZeRO's breakdown[4], training HBM falls into two buckets: Model States, which scale strictly with parameter count $\Phi$ (weights, gradients, and Adam optimizer states), and Residual States, which vary with sequence length, batch size, and allocator order (activations, temporary communication buffers, and unusable fragmented memory).

Modern pretraining pairs mixed precision (16-bit FP16/BF16 weights and gradients, $2\Phi + 2\Phi$) with an FP32 optimizer holding master weights plus Adam states ($K\Phi = 12\Phi$), giving a baseline of $\mathbf{16\Phi}$. During optimizer.step(), gradients are cast to FP32 ($4\Phi$): freeing or overwriting the 16-bit gradient buffer caps peak static HBM at $\mathbf{18\Phi}$, whereas holding both 16-bit and 32-bit gradient buffers simultaneously (e.g., under gradient accumulation or during Megatron's copy step) brings peak static state to $\mathbf{20\Phi}$:

Component Precision Bytes 7B 70B
Weights (model_param) BF16 / FP16 $2\Phi$ 14 GB 140 GB
Gradients BF16 / FP32 $2\Phi$ / $4\Phi$ 14 / 28 GB 140 / 280 GB
Master (main_param) FP32 $4\Phi$ 28 GB 280 GB
Adam Momentum $m$ FP32 $4\Phi$ 28 GB 280 GB
Adam Variance $v$ FP32 $4\Phi$ 28 GB 280 GB
Total (base / peak) Mixed Prec. $16\Phi$ ($18\Phi / 20\Phi$) 112–140 GB 1.12–1.40 TB

Even a 7B dense model consumes $112\text{–}140\text{ GB}$ of static state alone—already beyond a single 80 GB H100 before storing any activations.

The dominant Residual State is intermediate activations saved for backprop. Without activation checkpointing, a single Transformer layer (sequence length $s$, micro-batch $b$, hidden size $h$, attention heads $a$) holds[3]:

$$\text{Mem}_{\text{act/layer}} = s b h \left(34 + \frac{5 a s}{h}\right) \text{ bytes}$$

Plug in 70B dimensions ($h = 8192, a = 64, L = 80$) at just $b = 1, s = 4096$: activations swell to roughly $\mathbf{521\text{ GB}}$, more than $3.7\times$ the compute weights themselves.

Distributed Mixed Precision

Comparing the three floating-point formats across dynamic range (exponent bits) and near-1 resolution ($\varepsilon$ such that $1 + \varepsilon \gt 1$)[11]:

Format Bits (S-E-M) Bytes Magnitude Range Near-1 $\varepsilon$ Loss Scale
FP32 1-8-23 4 B $10^{-38} \sim 3\times 10^{38}$ $0.00000012$ None
FP16 1-5-10 2 B $5.95\times 10^{-8} \sim 65504$ $0.00097656$ Dynamic / Const
BF16 1-8-7 2 B $10^{-38} \sim 3\times 10^{38}$ $0.00781250$ None / Const

Pure 16-bit training fails for two concrete numerical reasons:

  • Rounding error: in $[2^{-3}, 2^{-2}]$, FP16's fixed step spacing is $2^{-13}$, so adding gradient $2^{-14}$ to weight $2^{-3}$ rounds straight back to $2^{-3}$, silently dropping the update (and BF16's coarser 7-bit mantissa needs FP32 master copies even more).
  • Gradient underflow: in late training, roughly 67% of activation gradients fall below $2^{-24}$ (the FP16 minimum normal positive limit) and vanish to zero—fixed by scaling loss by $S$ before backprop. Because BF16 keeps FP32's 8-bit exponent ("get into the right range first, then worry about precision"), it avoids underflow and skips dynamic loss scaling.

Meanwhile, small sensitivity-critical tensors such as LayerNorm/BatchNorm parameters and the scalar loss remain in FP32 throughout.

Three low-level mechanics in Megatron's Float16OptimizerWithFloat16Params explain how distributed mixed precision works in code:

  • main_grad does NOT mean "FP32 gradients": Under Megatron's local DDP (DDP_impl == "local", params_have_main_grad=True), parameters of the same dtype share a contiguous MemoryBuffer (accessed via tensor.main_grad instead of fragmented tensor.grad) so all_reduce runs in contiguous blocks. In FP16 runs, main_grad is still FP16 and is only cast via .float() inside _copy_model_grads_to_main_grads().
  • main_param = param.detach().clone().float(): Calling clone() alone leaves the master copy attached to autograd (requires_grad=True); calling detach() alone shares storage with FP16 weights. Chaining both isolates the FP32 copy before copying TP attributes, preserving PP-shared Embedding flags (param.shared), and repointing optimizer.param_groups.
  • DynamicGradScaler uses cumulative hysteresis: On inf/NaN, Megatron skips the optimizer update immediately, yet does not shrink $S$ on the first overflow. Instead it decrements cumulative _hysteresis_tracker and shrinks max(_scale * backoff_factor, min_scale) only once it reaches <= 0, while scaling up by growth_factor after growth_interval consecutive clean steps.
# Isolated FP32 master weight + cumulative hysteresis dynamic loss scaling [11]
main_param = param.detach().clone().float()
mpu.copy_tensor_model_parallel_attributes(main_param, param)

# DynamicGradScaler.update(found_inf): cumulative hysteresis backoff, consecutive growth
if found_inf:
    self._growth_tracker = 0
    self._hysteresis_tracker -= 1
    if self._hysteresis_tracker <= 0:
        self._scale = torch.max(self._scale * self.backoff_factor, self.min_scale)
else:
    self._growth_tracker += 1
    if self._growth_tracker == self.growth_interval:
        self._growth_tracker = 0
        self._hysteresis_tracker = self.hysteresis
        self._scale = self._scale * self.growth_factor

Inside Float16OptimizerWithFloat16Params.step(), each iteration executes six steps in order:

  1. Copy & cast gradients (_copy_model_grads_to_main_grads): Cast model_param.main_grad (or .grad) into FP32 main_param.grad.
  2. Unscale & global found_inf check (_unscale_main_grads_and_check_for_nan): Run torch._amp_foreach_non_finite_check_and_unscale_, synchronize self.found_inf via dist.all_reduce(MAX) across mpu.get_model_parallel_group() ($\text{TP} \times \text{PP}$), update grad_scaler, and skip the step cluster-wide if any rank overflowed.
  3. Global L2 clipping (clip_grad_norm): Compute $\|g\|_2 = \sqrt{\sum_{r \in \text{TP}} \sum_{s \in \text{PP}} \|g_{r,s}\|_2^2}$ (excluding TP-replicated params such as LayerNorm so they are not counted TP times) and scale $g \leftarrow \frac{c}{\|g\|_2}g$ when $\|g\|_2 \gt c$.
  4. Optional zero count: Log zero gradient counts via count_zeros() if enabled.
  5. Step optimizer (self.optimizer.step()): Update FP32 main_param (plus native FP32 parameters).
  6. Cast back (_copy_main_params_to_model_params): Copy updated FP32 master weights back into FP16/BF16 model_param via Apex's single-kernel _multi_tensor_copy_this_to_that.

Sharding & Collectives

Device Mesh Notation

Once training spans tens or thousands of GPUs, verbal descriptions of which slice lives where break down quickly. Naming the cluster axes—such as a 32-GPU 2D $\text{Mesh}(X=8, Y=4)$—lets us write sharding directly onto tensor subscripts[1]:

  • $A[I, J]$ (replicated): No subscript; every GPU holds the full tensor.
  • $A[I_X, J]$ (sharded on one axis): Dimension $I$ is cut across physical axis $X$; omitting $Y$ means shards are replicated along $Y$.
  • $A[I_X, J_Y]$ (2D sharded): Each card holds $\left(\frac{|I|}{|X|}, \frac{|J|}{|Y|}\right)$, shrinking memory by $1 / (|X|\cdot|Y|)$.
  • $A[I_{XY}, J]$ (flattened multi-axis shard): Both $X$ and $Y$ combine into one $|X|\cdot|Y|$-way partition along $I$.
  • $A[I_X, J_X]$ (invalid): Reusing the same hardware axis across two dimensions leaves only diagonal tiles.
  • $C[I, K]\{U_X\}$ (unreduced partial sum): Every card along $X$ holds a full-shaped $[I, K]$ partial buffer whose true value is the elementwise sum across $X$.
import jax, jax.numpy as jnp

mesh = jax.make_mesh(axis_sizes=(8, 4), axis_names=('X', 'Y'))
P = lambda *spec: jax.NamedSharding(mesh, jax.sharding.PartitionSpec(*spec))

# Cut E along the 4-GPU Y axis; replicate across the 8 X-axis groups
A = jnp.zeros((2048, 8192), dtype=jnp.bfloat16, device=P('Y', None))

Consider a concrete BF16 matrix $A[2048, 8192]$ on $\text{Mesh}(X=8, Y=4)$. Writing $A[E_Y, F]$ cuts dimension $E$ across $Y$ alone ($4$ cards), so each GPU holds $2048 / 4 = 512$ rows ($512 \times 8192 \times 2\text{ B} = \mathbf{8\text{ MiB}}$) with $8$ identical replicas along $X$. Spreading rows across all 32 cards ($64$ rows each, $\mathbf{1\text{ MiB}}$) requires two subscripts: $A[E_{XY}, F]$. An easy sanity check confirms this: an AllGather_Y over 4 cards turns $512 \times 4$ cleanly back into $2048$ rows, whereas $64 \times 4 = 256$ falls short by the missing $8\times$ along $X$.

Fig. 1. Local slice shape, HBM footprint, and replica groups for A[E_Y, F], A[E_XY, F], and A[E_Y, F_X] on a 32-GPU 8×4 mesh (click any cell to inspect).

The Four Collectives

Here All refers to every GPU sharing a named mesh axis (e.g., the 4 ranks at a fixed $X$). Separating the four collectives comes down to two questions: Does data get summed elementwise? And does each card end with the whole tensor or a slice?

Collective Signature Mechanics Typical Usage
AllGather $[A_X, B] \to [A, B]$ Collect shards and concatenate along unsharded axis FSDP / ZeRO-3 reconstructing weights before compute
ReduceScatter $[A, B]\{U_X\} \to [A_X, B]$ Sum elementwise, then scatter disjoint slices Summing grads so each rank keeps its optimizer slice
AllReduce $[A_X, B]\{U_Y\} \to [A_X, B]$ Sum elementwise and replicate full result ($\text{RS} + \text{AG}$) DDP gradient sync, Megatron Row-Parallel output
All-to-All $[E_Y, F] \to [E, F_Y]$ Split local data into $N$ chunks; send chunk $j$ to rank $j$ MoE token routing, row-split to col-split reshard

Running all four from the exact same 2-GPU initial state ($\text{GPU}_0 = [1, 2]$, $\text{GPU}_1 = [3, 4]$):

Operation GPU 0 Output GPU 1 Output Size Change
Initial input $[1, 2]$ $[3, 4]$ Each GPU holds half ($1\times$)
AllGather $[1, 2, 3, 4]$ $[1, 2, 3, 4]$ Concatenated without summing ($2\times$)
ReduceScatter $[1+3] = [4]$ $[2+4] = [6]$ Summed then split ($0.5\times$)
AllReduce $[4, 6]$ $[4, 6]$ ReduceScatter followed by AllGather ($1\times$ full)
All-to-All $[1, 3]$ $[2, 4]$ Chunk 0 to GPU 0, chunk 1 to GPU 1 ($1\times$)

Note the order: AllReduce is ReduceScatter + AllGather, not AllGather + Scatter. Plain Scatter only splits data without adding partial values like $1+3$ and $2+4$. Whenever the next consumer already operates on shards, stopping after ReduceScatter saves half the traffic.

Fig. 2. Animated value flow across all four collectives from $[1, 2]$ and $[3, 4]$, showing AllReduce = ReduceScatter + AllGather.

To compare AllGather against All-to-All, reuse the 4-GPU $Y$ group holding $A[E_Y, F]$ ($512 \times 8192$, $8\text{ MiB}$ per rank):

Operation Per-Rank Send Pattern Output Shape Local HBM
$\text{AllGather}_Y$ Broadcast full 512 rows to all 3 peers $[2048, 8192]$ $32\text{ MiB}$ ($4\times$)
$\text{AllToAll}_Y$ Split 8192 cols into 4 pieces; send $j$-th to rank $j$ $[2048, 2048]$ $8\text{ MiB}$ ($1\times$)

Under AllToAll_Y, GPU 0 receives four $512 \times 2048$ blocks (all rows, first quarter of columns) and stacks them vertically—converting a row-sharded matrix into a column-sharded one with zero memory growth.

Fig. 3. Contrasting AllGather_Y (reconstructing full $32\text{ MiB}$ tensors) against AllToAll_Y (transposing row-shards to col-shards at constant $8\text{ MiB}$).

One-line rule: Ask what the next op lacks:

  • Full tensor from disjoint shards $\implies$ AllGather
  • Full replicated sum from partial buffers $\{U_X\}$ $\implies$ AllReduce
  • Sharded slice of a sum from $\{U_X\}$ $\implies$ ReduceScatter
  • Resharding across axes or routing MoE tokens $\implies$ All-to-All

Ring Cost & Duality

Let $V$ be total unsharded tensor bytes, $|X|$ be rank count along an axis, $W_{\text{ici}}$ be per-link bidirectional bandwidth, and $T_{\min}$ be per-hop latency. On a bidirectional ring, each GPU simultaneously sends $\frac{V}{|X|}$ chunks left and right over $\lfloor |X|/2 \rfloor$ hops; $|X|$ cancels out of the bandwidth term, and splitting across $N_{\text{axes}}$ orthogonal links scales bandwidth linearly:

$$T_{\text{AG / RS}} = \max\!\left(\frac{T_{\min} \cdot \sum_i |X_i|}{2},\; \frac{V}{W_{\text{ici}} \cdot N_{\text{axes}}}\right)$$

For large tensors (bandwidth-bound), AllGather and ReduceScatter cost $\frac{V}{W_{\text{ici}}}$ regardless of group size $|X|$, while AllReduce runs both in sequence and costs $\frac{2V}{W_{\text{ici}}}$.

On the same bidirectional ring, All-to-All takes only one quarter the time of AllGather: AllGather sends every card's entire shard across all hops ($\propto D^2$), whereas All-to-All delivers each rank-specific $\frac{V}{D^2}$ piece only up to its destination distance ($1 + 2 + \dots + D/2$ in each direction), cutting total traffic by $4\times$. Generalized to a $D$-dimensional mesh $(A \times B \times \dots)$ of $N$ cards:

$$T_{\text{AllToAll}} = \frac{V \cdot \max(A, B, \dots)}{4 \cdot N \cdot W_{\text{ici}}} \xrightarrow{\text{1D Ring } (N = A)} \frac{V}{4 \cdot W_{\text{ici}}} = \frac{1}{4}\, T_{\text{AllGather}}$$
Fig. 4. Four-GPU bidirectional ring packet hops alongside large-tensor bandwidth costs showing why All-to-All is $4\times$ cheaper than AllGather.

In autograd, collectives obey an exact transpose duality. Writing broadcast and reduce with an all-ones vector $\mathbf{u} \in \mathbb{R}^p$ and Kronecker product $\otimes$:

$$\begin{aligned} \text{broadcast} &= \mathbf{u} \otimes \mathbf{I}_n, & \text{reduce} &= \mathbf{u}^\top \otimes \mathbf{I}_n = \text{broadcast}^\top \\ \text{AllGather} &= \text{broadcast} \otimes \mathbf{I}_p, & \text{ReduceScatter} &= \text{reduce} \otimes \mathbf{I}_p = \text{AllGather}^\top \end{aligned}$$

Since backprop through any linear map $\mathcal{A}$ multiplies by $\mathcal{A}^\top$, a forward AllGather differentiates automatically into a backward ReduceScatter—the exact algebraic backbone of FSDP and Sequence Parallelism.

Sharded Matmul

Every distributed linear layer evaluates $A[I, J] \cdot B[J, K] \to C[I, K]$ along contracting axis $J$, falling into one of four cases:

  1. Neither side cuts $J$: $A[I_X, J] \cdot B[J, K_Y] \to C[I_X, K_Y]$. Zero forward communication—covers Data Parallel forward and Megatron Column-Parallel layers.
  2. Only one side cuts $J$: $A[I, J_X] \cdot B[J, K]$. Usually preceded by AllGather_X (FSDP), giving time $\max(\frac{2BDF}{C}, \frac{2DF}{W_{\text{ici}}})$, where communication hides behind compute once batch tokens $B > C / W_{\text{ici}}$. In real training activations are already sharded on the same axis $X$, so computing locally and calling AllReduce is blocked anyway.
  3. Both sides cut $J$ on the same axis $X$: $A[I, J_X] \cdot_{\text{LOCAL}} B[J_X, K] \to C[I, K]\{U_X\}$. Ranks multiply locally to emit partial sums $\{U_X\}$, then finish with AllReduce_X (Megatron Row-Parallel) or ReduceScatter to preserve sharding (DP weight gradients / Sequence Parallelism).
  4. Outer dimensions collide on $X$: $A[I_X, J] \cdot B[J, K_X]$. Invalid directly because rank $i$ only holds diagonal blocks $(I_i, K_i)$; call AllGather_X on the smaller factor first.

High-throughput kernels pipeline Cases 2 and 3 into a Collective Matmul, tiling Tensor Core MMA instructions alongside ring transfers of the next chunk.

Fig. 5. Block-matrix view of the four sharded matmul cases determining zero-comm execution, pre-AllGather, or post-AllReduce / ReduceScatter.

Data Parallelism & ZeRO

DP and DDP

Pure Data Parallelism runs Case 1 forward (sharded batch $B_{\text{DP}}$) and Case 3 backward on weight gradients $dW = X[B_{\text{DP}}, D]^\top \cdot dY[B_{\text{DP}}, F]$, contracting along $B_{\text{DP}}$ to produce $\{U_{\text{DP}}\}$.

Under the classic Parameter Server DP model[8], $N$ compute Workers push local gradients to a single Server (often GPU 0 on a single node) and pull back averaged updates. Total cluster traffic is $N\Phi\text{ (Server)} + N\Phi\text{ (Workers)} = 2N\Phi$, so the Server scales linearly in $N$ and saturates its NIC across nodes (though multi-Server setups shard parameters across servers).

To keep Workers busy while the Server syncs, PS frameworks also support asynchronous bounded-delay updates—letting a Worker advance iter 11 on stale weights while iter 10 syncs (e.g., enforcing iter 10–11 applied before iter 12 under 1-bounded delay), trading slower SGD convergence for hardware utilization.

DDP eliminates the central Server using Ring-AllReduce: each GPU slices its gradients (not input batch) into $N$ chunks of $\Phi/N$ and runs $N-1$ hops of ReduceScatter ($(N-1)\frac{\Phi}{N}$ send/GPU) followed by $N-1$ hops of AllGather ($(N-1)\frac{\Phi}{N}$ send/GPU), fixing per-GPU send volume at a rank-independent $2\frac{N-1}{N}\Phi \approx \mathbf{2\Phi}$.

In PyTorch DDP, one process per GPU avoids Python's GIL while a C++ Reducer packs parameters in reverse topological order into ~25 MB buckets so NCCL overlaps asynchronous ring reductions behind ongoing backward passes.

ZeRO and FSDP

DDP balances bandwidth yet keeps $N$ redundant $16\Phi$ copies of model states and repeats identical Adam updates on every card. DeepSpeed ZeRO[4][9] (and PyTorch FSDP) shards optimizer states ($P_{os}$), gradients ($P_g$), and parameters ($P_p$) across $N_d$ DP ranks ("consume and discard, re-gather when needed"):

Tracking our 2-GPU gradients $g_0=[1,2]$ and $g_1=[3,4]$: DDP runs AllReduce(g) so both GPUs hold $[4,6]$ and update all weights; ZeRO-3 / FSDP permanently stores only $W_0, \text{OS}_0$ on GPU 0, runs AllGather(W) right before computing each layer (and frees peer shards immediately after), then executes ReduceScatter(g) so GPU 0 receives only $[4]$ to update $W_0$ while GPU 1 gets $[6]$ for $W_1$:

Strategy Sharded Static HBM 7.5B (N=64) Comm / Rank
DDP None $16\Phi$ 120 GB $2\Phi$ ($1\times$)
ZeRO-1 Optimizer States $P_{os}$ $4\Phi + \frac{12\Phi}{N_d}$ 31.4 GB $2\Phi$ (paper $3\Phi$)
ZeRO-2 $P_{os}$ + Gradients $P_g$ $2\Phi + \frac{14\Phi}{N_d}$ 16.6 GB $2\Phi$ ($1\times$)
ZeRO-3 / FSDP $P_{os}$ + $P_g$ + Params $P_p$ $\frac{16\Phi}{N_d}$ 1.9 GB $3\Phi$ ($1.5\times$)

ZeRO-1 $3\Phi$ vs. $2\Phi$ traffic & gradient accumulation: conceptually the ZeRO paper counts Stage-1 traffic as full-grad AllReduce ($2\Phi$) + AllGather(W) ($\Phi$) = $3\Phi$, whereas modern implementations swap AllReduce(G) for ReduceScatter(G) ($\Phi$) to stay at $2\Phi$. Still, keeping full gradients in Stage 1 saves $m-1$ sync steps under $m$-step gradient accumulation compared with Stage 2.

PyTorch Memory Snapshots on 2 GPUs reveal two more ZeRO-1 implementation details:

  • In-place FP16 weight refresh: updated weights write directly via fp16._copy_(fp32), reusing persistent GPU buffers.
  • Lazy Adam state allocation: momentum $m, v$ tensors are allocated only on the first clean step without NaN/inf overflow.

ZeRO is best summarized as "model-parallel in form, data-parallel in substance": even Stage 3 processes disjoint micro-batches with transiently gathered full layer weights (freeing peer shards immediately after each layer).

Alongside model states, ZeRO-R targets residual memory from three angles:

  • $P_a$ (partitioned activations): shards activations replicated across TP ranks (pure DP activations are already batch-split).
  • $C_B$ (constant-size buffers): batches small cross-GPU slices up to a fixed buffer cap to keep interconnect bandwidth saturated.
  • $M_D$ (memory defragmentation): consolidates fragmented HBM left behind by mixed short- and long-lived tensors.

When HBM remains tight, ZeRO-Offload / ZeRO-Infinity keeps compute-heavy forward/backward passes on GPUs while offloading large, low-FLOP FP32 master weights and optimizer steps to CPU RAM.

Fig. 6. Side-by-side step lifecycle of DDP vs. ZeRO-3 / FSDP alongside the $16\Phi \to 2\Phi$ per-GPU HBM reduction at $N=8$.

Tensor Parallelism

MLP, Attention & Vocab

ZeRO reduces static state, not intra-layer activations. Tensor Parallelism (TP)[2][10] splits each linear matmul $Y = XW$ ($X:[b,s,h], W:[h,h']$) in one of two ways:

  • Row-split (along $h$): forces a matching column-split on $X$, sums partial outputs via $g$ in forward, and replicates/concatenates input gradients via $f$ in backward.
  • Column-split (along $h'$): replicates $X$, concatenates output slices $[Y_1,Y_2]$, and relies on $f$ in backward to AllReduce $\partial\mathcal{L}/\partial X|_1 + \partial\mathcal{L}/\partial X|_2$.

Megatron-LM pairs Column-Parallel then Row-Parallel linears so nonlinearities and softmax run with zero communication:

  • MLP: Column-split $A = [A_0, A_1]$ and row-split $B = \begin{bmatrix} B_0 \\ B_1 \end{bmatrix}$. Because $\text{GeLU}(Y_1+Y_2) \neq \text{GeLU}(Y_1)+\text{GeLU}(Y_2)$ whereas $\text{GeLU}([Y_1,Y_2]) = [\text{GeLU}(Y_1),\text{GeLU}(Y_2)]$, column-splitting $A$ avoids syncing before GeLU and uses only 1 forward + 1 backward AllReduce ($4bsh$ total): $$Z = \text{AllReduce}\!\left(\sum_i \text{GeLU}(X A_i)\, B_i\right)$$
  • Multi-Head Attention: Split $W^Q, W^K, W^V$ along k_dim by heads ($\text{num\_heads} \bmod \text{TP} = 0$) and row-split $W^O$, also costing $4bsh$—giving 4 AllReduces ($8bsh$) per Transformer layer. Adding Sequence Parallelism (SP)[3] splits these into AllGather and ReduceScatter at identical traffic while eliminating TP-replicated LayerNorm/Dropout activations.
  • Vocab Embedding & VocabParallelCrossEntropy: Position embeddings $[s_{\max}, h]$ stay replicated while word embeddings $[V, h]$ split along $V$. For token IDs [0,212,7,9] across $[0,150)$ and $[150,299)$, each GPU masks out-of-range IDs to zero ([ok,0,ok,ok] + [0,ok,0,0]) and runs one AllReduce. When $\text{PP} \gt 1$, input and output tie the same word embedding, so Megatron must run an extra AllReduce across the first and last stages before stepping. On output logits $[b,s,V/\text{TP}]$, exact Log-Sum-Exp normalization avoids gathering $[b,s,V]$ by syncing only three lightweight $[b,s]$ scalars: $$\begin{aligned} m_{\text{global}} &= \text{AllReduce}_{\max}\!\left(\max_{j \in V_i} z_{i,j}\right), \qquad z'_{i,j} = z_{i,j} - m_{\text{global}} \\ S_{\text{global}} &= \text{AllReduce}_{\text{sum}}\!\left(\sum_{j \in V_i} e^{z'_{i,j}}\right), \qquad z'_{\text{target}} = \text{AllReduce}_{\text{sum}}\!\left(\mathbb{I}[y \in V_i]\, z'_{i,y}\right) \\ \mathcal{L} &= \log S_{\text{global}} - z'_{\text{target}} \end{aligned}$$

Why keep TP intra-node and push DP inter-node? Two factors drive the topology choice:

  • Per-layer volume: TP sends $8bsh$ vs. DP's $2h^2$—canceling $h$ compares $[b\cdot s]$ against $[h]$ (both $\sim 10^5$, with TP typically larger at long sequence lengths).
  • The decisive bottleneck—backward blocking: DP launches gradient AllReduce asynchronously while earlier layers keep differentiating, whereas TP stalls every single layer until AllReduce returns $\partial\mathcal{L}/\partial X$.

Across 32× DGX-2H nodes (512× V100-32GB) training GPT-2 (1.2B→8.3B), pure TP scales at 100%→95%→82%→77% from 1 to 8 GPUs (capping at 8/16 GPUs per node due to activation HBM), while pairing intra-node $\text{TP}=8$ with inter-node $\text{DP}=64$ reaches 15,100 TFLOPs out of $39 \times 512 = 19{,}968\text{ TFLOPs}$ ideal—76% weak scaling efficiency.

Megatron Internals

Three low-level mechanics inside Megatron's mpu (parallel_state) module keep 3D parallelism fast and numerically identical to single-GPU training:

1. 3D process group layout (initialize_model_parallel): Megatron orders global ranks from innermost TP → middle DP → outermost PP. With 16 GPUs across two 8-GPU nodes ($\text{TP}=2, \text{PP}=2, \text{DP}=4$), it constructs four orthogonal process groups:

  • TP groups (stride $1$, eight 2-GPU pairs): [0,1], [2,3], ..., [14,15]—adjacent ranks stay inside the same intra-node NVLink switch.
  • DP groups (stride $\text{TP}=2$, four 4-GPU groups): [0,2,4,6] & [1,3,5,7] on Stage 0 and [8,10,12,14] & [9,11,13,15] on Stage 1, syncing matching weight shards across replicas.
  • PP groups (stride $\text{world}/\text{PP}=8$, eight 2-stage pipelines): [0,8], [1,9], ..., [7,15], chaining nodes across pipeline stages.
  • Model-parallel groups $\text{MP} = \text{TP} \times \text{PP}$ (four 4-GPU replicas): [0,1,8,9], [2,3,10,11], [4,5,12,13], [6,7,14,15], used for the global found_inf consensus check and global L2 clip_grad_norm.
Fig. 7. The four orthogonal process groups generated by initialize_model_parallel for 16 GPUs ($\text{TP}=2, \text{PP}=2, \text{DP}=4$).

In code, each rank computes the exact same sorted lists and calls torch.distributed.new_group(ranks) in lockstep—every process must call new_group in identical global order even when rank not in ranks so collective group creation never deadlocks:

def initialize_model_parallel(tp_size: int = 2, pp_size: int = 2):
    world_size = dist.get_world_size()          # e.g., 16 = TP(2) * PP(2) * DP(4)
    rank = dist.get_rank()
    dp_size = world_size // (tp_size * pp_size)
    num_tp_groups = world_size // tp_size       # 8 TP groups
    num_pp_groups = world_size // pp_size       # 8 PP pipelines

    # 1. DP groups (stride = tp_size): [0,2,4,6], [1,3,5,7], ...
    for i in range(pp_size):
        start = i * num_pp_groups
        end = (i + 1) * num_pp_groups
        for j in range(tp_size):
            ranks = list(range(start + j, end, tp_size))
            group = dist.new_group(ranks)
            if rank in ranks:
                _DATA_PARALLEL_GROUP = group

    # 2. Tensor Parallel groups (contiguous, stride = 1): [0,1], [2,3], ..., [14,15]
    for i in range(num_tp_groups):
        ranks = list(range(i * tp_size, (i + 1) * tp_size))
        group = dist.new_group(ranks)
        if rank in ranks:
            _TENSOR_MODEL_PARALLEL_GROUP = group

    # 3. Pipeline Parallel groups (stride = num_pp_groups = 8): [0,8], [1,9], ..., [7,15]
    for i in range(num_pp_groups):
        ranks = list(range(i, world_size, num_pp_groups))
        group = dist.new_group(ranks)
        if rank in ranks:
            _PIPELINE_MODEL_PARALLEL_GROUP = group

    # 4. Full Model Parallel groups (TP x PP): [0,1,8,9], [2,3,10,11], [4,5,12,13], [6,7,14,15]
    for i in range(dp_size):
        ranks = [i * tp_size + p * num_pp_groups + t
                 for p in range(pp_size) for t in range(tp_size)]
        group = dist.new_group(ranks)
        if rank in ranks:
            _MODEL_PARALLEL_GROUP = group

2. Conjugate autograd ops f/g and linear wiring: ColumnParallelLinear applies f at its input (keeping gather_output=False so GeLU runs on sharded activations), and RowParallelLinear (input_is_parallel=True) closes with g. Because forward sends replicated $X$ into all $A_i$, chain-rule backprop $\frac{\partial\mathcal{L}}{\partial X} = \sum_i \frac{\partial\mathcal{L}}{\partial Y_i}A_i^\top$ requires summing gradients inside f.backward:

class _CopyToModelParallelRegion(torch.autograd.Function):
    """Op f: Identity in forward, AllReduce in backward (ReduceScatter under SP)."""
    @staticmethod
    def forward(ctx, input_): return input_
    @staticmethod
    def backward(ctx, grad_output): return _reduce(grad_output)

class _ReduceFromModelParallelRegion(torch.autograd.Function):
    """Op g: AllReduce in forward (ReduceScatter under SP), Identity in backward."""
    @staticmethod
    def forward(ctx, input_): return _reduce(input_)
    @staticmethod
    def backward(ctx, grad_output): return grad_output

# ParallelMLP flow (zero comm across GeLU):
# x -> f -> ColumnParallelLinear(gather_output=False) -> GeLU
#   -> RowParallelLinear(input_is_parallel=True) -> g -> out
Fig. 8. Forward/backward signal flow through Megatron's conjugate f / g operators, with a toggle for Sequence Parallelism (TP + SP).

3. CudaRNGStatesTracker: Dropout masks inside TP-sharded activations use a distinct rank-seeded state (model-parallel-rng, seeded with base_seed + 2718 + tp_rank), whereas Dropout on replicated residual paths exits the context so every TP rank shares the identical base mask:

with get_cuda_rng_tracker().fork('model-parallel-rng'):
    # Sharded activations (e.g., inside attention): independent masks per tp_rank
    attention_probs = self.attention_dropout(attention_probs)
# Restored base_seed on replicated residual path: identical mask across TP ranks
hidden_states = self.hidden_dropout(output)

Because TP's blocking collectives sit directly on the layer critical path and cannot hide behind asynchronous buckets, practical clusters keep $\text{TP} \le 8$ within a single NVLink node.

Pipeline Parallelism

GPipe and 1F1B

When a model outgrows 8 GPUs, pushing TP over cross-node Ethernet/InfiniBand tanks utilization. Pipeline Parallelism (PP) slices the model vertically into $P$ stages connected only by lightweight P2P send/recv transfers of boundary activations.

Feeding one unsplit batch leaves $P-1$ stages idle ($\frac{P-1}{P} = 75\%$ bubble at $P=4$). GPipe[5] splits each mini-batch into $M$ micro-batches; assuming equal forward and backward step times, idle bubble overhead shrinks to:

$$r_{\text{GPipe}} = \frac{P - 1}{M + P - 1}$$

Taking $M \ge 4P$ drops bubble time below $16\%$.

Stashing full intermediate activations for all $M$ micro-batches in flight would still blow HBM, so GPipe pairs scheduling with activation checkpointing (re-materialization): forward stores only stage boundary tensors and recomputes internal layers on the backward pass, trading ~33% extra FLOPs for peak memory $\mathcal{O}\!\left(N + \frac{L}{P}\cdot\frac{N}{M}\right)$.

Even with recomputation, GPipe holds all $M$ micro-batches at Stage 0 until every forward pass completes, coupling memory to $\mathcal{O}(M)$. Pipelining schedules break that coupling:

  • 1F1B: alternates one forward step and one backward step as soon as the pipeline fills, locking peak buffered micro-batches at $\mathcal{O}(P)$ while keeping the same bubble ratio.
  • Interleaved 1F1B: assigns $v$ non-contiguous layer chunks per GPU to shrink the bubble to $\frac{P-1}{vM + P - 1}$ in exchange for $v\times$ tiny P2P transfers.

Because first and last stages also carry Embeddings and output heads, practical setups place fewer Transformer layers at both ends to balance stage compute and memory.

Fig. 9. Timeline sweep at $P=4, M=8$ comparing Naive, GPipe, 1F1B, and Interleaved ($v=2$) alongside Stage 0 live buffered activation count.

MoE & Expert Parallelism

Routing & Load Balance

Mixture-of-Experts (MoE) decouples parameter count from per-token FLOPs by replacing each FFN with $E$ expert networks and dynamically routing every token via $P = \text{Softmax}(x W_g)$ to its top-$k$ experts.

To keep static tensor shapes on GPUs, each expert caps its buffer at a fixed Expert Capacity[6]:

$$C = \left\lceil \frac{N_{\text{tokens}} \cdot k}{E} \cdot \text{capacity\_factor} \right\rceil$$

With capacity_factor typically in $1.0\text{–}1.5$, overflow tokens bypass the expert via the residual connection (dropped), while underfilled slots are zero-padded.

Because unregularized routers collapse onto one or two popular experts, MoE adds a differentiable load-balancing loss ($\alpha$ scaled by $E$ so magnitude stays invariant to expert count):

$$\mathcal{L}_{\text{aux}} = \alpha \cdot E \sum_{i=1}^{E} f_i \cdot P_i$$

Here $f_i$ is the actual fraction of tokens dispatched to expert $i$ and $P_i$ is mean router probability; by Cauchy-Schwarz this attains its unique global minimum $\alpha$ iff $f_i = P_i = 1/E$.

All-to-All & EP Groups

Whereas ZeRO-3 keeps tokens stationary and moves weights via AllGather, Expert Parallelism (EP)[7] pins massive expert weights on fixed GPUs and ships lightweight tokens over two All-to-All passes—capitalizing on its $4\times$ bandwidth advantage on a ring.

On two GPUs where GPU 0 hosts $E_0$ and holds $[a, b]$, and GPU 1 hosts $E_1$ and holds $[c, d]$ (routing $a,c \to E_0$ and $b,d \to E_1$): Pass 1 (Dispatch) exchanges $b$ and $c$ so each GPU computes its local experts; Pass 2 (Combine) returns $[a', b']$ and $[c', d']$ to their origin ranks for the next Attention block:

class _AllToAll(torch.autograd.Function):
    @staticmethod
    def forward(ctx, group, x):
        ctx.group = group
        out = torch.empty_like(x)
        dist.all_to_all_single(out, x, group=group)
        return out

    @staticmethod
    def backward(ctx, *grad_out):
        # All-to-All is self-adjoint
        return (None, _AllToAll.apply(ctx.group, *grad_out))

dispatched = torch.einsum("sec,sm->ecm", dispatch_mask, x)
dispatched = _AllToAll.apply(ep_group, dispatched)          # Pass 1: dispatch
expert_out = experts(dispatched)
combined = _AllToAll.apply(ep_group, expert_out)            # Pass 2: combine
output = torch.einsum("sec,ecm->sm", combine_weights, combined)
Fig. 10. Two-pass All-to-All token routing cycle and a skewed 4-GPU CF=1.0 example showing capacity drops.

In DeepSpeed-Megatron, shared parameters (Attention, Embeddings, LayerNorm, Router) replicate across all DP ranks while experts shard along $\text{EP}$. With 8 DP GPUs, 4 experts, and $\text{EP}=4$, the cluster forms $2$ full expert replicas—splitting DP cleanly as $\text{DP}(8) = \text{EP}(4) \times \text{EDP}(2)$:

  • EP groups ({0,1,2,3}, {4,5,6,7}): Route tokens via two All-to-All passes.
  • EDP groups ({0,4}, {1,5}, {2,6}, {3,7}): Sync gradients via AllReduce strictly across identical expert replicas—never averaging $E_0$ with $E_1$.
  • Global DP group ({0..7}): Sync all shared non-expert parameters across every rank.

Inside each expert FFN, intra-node TP can still be nested normally, maintaining $\text{world\_size} = \text{PP} \times \text{DP} \times \text{TP}$ with $\text{DP} = \text{EP} \times \text{EDP}$.

Fig. 11. Orthogonal process groups on an 8-GPU MoE cluster (DP(8) = EP(4) × EDP(2)) separating token dispatch, expert replica sync, and shared-weight DP sync.

4D Hybrid Parallelism

Putting every strategy side by side makes cluster topology design mechanical:

Strategy Sharded Axis Communication Memory Saved Target Link
DDP Batch $B_{\text{DP}}$ Async bucketed AllReduce Batch activations $/ \text{DP}$ Outer IB / RoCE
ZeRO-1 / 2 +$P_{os}$ / $P_g$ Grad RS + updated weight AG OS/G $/ \text{DP}$ at $2\Psi$ total Outer IB / RoCE
ZeRO-3 / FSDP +$P_p$ Weights Layerwise AG(W) + RS(g), total $3\Psi$ Static state $16\Phi / \text{DP}$ NVLink / fast IB
TP / TP+SP Col / Row (+$S_{\text{TP}}$) 2 blocking syncs fwd & bwd Weights & all acts $/ \text{TP}$ Intra-node NVLink $(\le 8)$
PP (1F1B) Layers ($P$ stages) Light P2P send/recv States $/ P$, acts $\mathcal{O}(P)$ Cross-node IB
EP (MoE) Experts ($E_{\text{EP}}$) 2 All-to-All per MoE layer Expert states $/ \text{EP}$ NVLink / full-bisection IB

On 64 × H100-80GB (8 nodes of 8 GPUs), map the heaviest communication onto the fastest fabric first:

  1. 70B dense ($16\Phi \approx 1.12\text{ TB}$): Lock $\text{TP}=8$ with Sequence Parallelism inside each 8-GPU NVLink node ($140\text{ GB}$/rank static); add cross-node $\text{PP}=2$ with 1F1B ($M \ge 8$, ~11% bubble) to cut static state to $70\text{ GB}$; use the remaining $\text{DP}=64/(8 \times 2)=4$ lanes with zero-overhead ZeRO-1, bringing per-GPU static HBM to $\frac{4\Phi}{16} + \frac{12\Phi}{64} = 17.5 + 13.1 = \mathbf{30.6\text{ GB}}$ and leaving ~50 GB free for long-context activations.
  2. 64-GPU MoE: Pair intra-node $\text{TP}=4$ with $\text{PP}=2$, then split $\text{DP}=8$ into $\text{EP}=4, \text{EDP}=2$ so 4-GPU groups exchange tokens via All-to-All, 2-way EDP syncs matching experts, and global DP + ZeRO-1 updates shared Attention and Router weights.

Citation

If you reference these notes or interactive animations, please cite:

Chaoran Feng (2026). "Distributed LLM Training: Sharding, Collectives, and Parallelism". Chaoran Feng's Blog. https://www.falcary.com/blogs/distributed-training/

@article{feng2026distributed,
  title   = {Distributed LLM Training: Sharding, Collectives, and Parallelism},
  author  = {Feng, Chaoran},
  year    = {2026},
  month   = {October},
  url     = {https://www.falcary.com/blogs/distributed-training/}
}

References

  1. How to Scale Your Model, Part 3: Sharded Matrices and How to Multiply Them. Austin, Bradbury, Blake, Chou, Douglas, Frostig, Levskaya, Pope, et al., Google DeepMind, 2025.
  2. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. Shoeybi et al., NVIDIA, 2019 (arXiv:1909.08053).
  3. Reducing Activation Recomputation in Large Transformer Models. Korthikanti et al., NVIDIA, 2022 (arXiv:2205.05198).
  4. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. Rajbhandari et al., Microsoft DeepSpeed, SC 2020 (arXiv:1910.02054).
  5. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. Huang et al., Google Brain, NeurIPS 2019 (arXiv:1811.06965).
  6. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. Lepikhin et al., Google Brain, ICLR 2021 (arXiv:2006.16668).
  7. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. Rajbhandari et al., ICML 2022 (arXiv:2201.05596).
  8. 图解大模型训练之:数据并行上篇(DP, DDP 与 ZeRO),猛猿(公众号:大猿搬砖简记),知乎。
  9. 图解大模型训练之:数据并行下篇(DeepSpeed ZeRO,零冗余优化),猛猿(公众号:大猿搬砖简记),知乎。
  10. 图解大模型训练之:张量模型并行(TP),Megatron-LM,猛猿(公众号:大猿搬砖简记),知乎。
  11. 图解大模型训练系列之:Megatron 源码解读 3,分布式混合精度训练,猛猿(公众号:大猿搬砖简记),知乎。