I recently reviewed and reorganized my notes on distributed LLM training, drawing from Part 3 of Google's JAX Scaling Book[1], 猛猿's four illustrated walkthroughs on DP/DDP[8], DeepSpeed ZeRO[9], Megatron-LM tensor parallelism[10], and Megatron mixed-precision internals[11], alongside the foundational papers[2][5][6][7]. The sections walk straight through HBM memory and mixed precision, device meshes and the four collectives, DDP/ZeRO, TP, PP, and MoE before sizing a 64-GPU cluster.
Memory & Mixed Precision
Where Memory Goes
Following ZeRO's breakdown[4], training HBM falls into two buckets: Model States, which scale strictly with parameter count $\Phi$ (weights, gradients, and Adam optimizer states), and Residual States, which vary with sequence length, batch size, and allocator order (activations, temporary communication buffers, and unusable fragmented memory).
Modern pretraining pairs mixed precision (16-bit FP16/BF16 weights and gradients, $2\Phi + 2\Phi$) with an FP32 optimizer holding master weights plus Adam states ($K\Phi = 12\Phi$), giving a baseline of $\mathbf{16\Phi}$. During optimizer.step(), gradients are cast to FP32 ($4\Phi$): freeing or overwriting the 16-bit gradient buffer caps peak static HBM at $\mathbf{18\Phi}$, whereas holding both 16-bit and 32-bit gradient buffers simultaneously (e.g., under gradient accumulation or during Megatron's copy step) brings peak static state to $\mathbf{20\Phi}$:
| Component | Precision | Bytes | 7B | 70B |
|---|---|---|---|---|
Weights (model_param) |
BF16 / FP16 | $2\Phi$ | 14 GB | 140 GB |
| Gradients | BF16 / FP32 | $2\Phi$ / $4\Phi$ | 14 / 28 GB | 140 / 280 GB |
Master (main_param) |
FP32 | $4\Phi$ | 28 GB | 280 GB |
| Adam Momentum $m$ | FP32 | $4\Phi$ | 28 GB | 280 GB |
| Adam Variance $v$ | FP32 | $4\Phi$ | 28 GB | 280 GB |
| Total (base / peak) | Mixed Prec. | $16\Phi$ ($18\Phi / 20\Phi$) | 112–140 GB | 1.12–1.40 TB |
Even a 7B dense model consumes $112\text{–}140\text{ GB}$ of static state alone—already beyond a single 80 GB H100 before storing any activations.
The dominant Residual State is intermediate activations saved for backprop. Without activation checkpointing, a single Transformer layer (sequence length $s$, micro-batch $b$, hidden size $h$, attention heads $a$) holds[3]:
$$\text{Mem}_{\text{act/layer}} = s b h \left(34 + \frac{5 a s}{h}\right) \text{ bytes}$$Plug in 70B dimensions ($h = 8192, a = 64, L = 80$) at just $b = 1, s = 4096$: activations swell to roughly $\mathbf{521\text{ GB}}$, more than $3.7\times$ the compute weights themselves.
Distributed Mixed Precision
Comparing the three floating-point formats across dynamic range (exponent bits) and near-1 resolution ($\varepsilon$ such that $1 + \varepsilon \gt 1$)[11]:
| Format | Bits (S-E-M) | Bytes | Magnitude Range | Near-1 $\varepsilon$ | Loss Scale |
|---|---|---|---|---|---|
FP32 |
1-8-23 |
4 B | $10^{-38} \sim 3\times 10^{38}$ | $0.00000012$ | None |
FP16 |
1-5-10 |
2 B | $5.95\times 10^{-8} \sim 65504$ | $0.00097656$ | Dynamic / Const |
BF16 |
1-8-7 |
2 B | $10^{-38} \sim 3\times 10^{38}$ | $0.00781250$ | None / Const |
Pure 16-bit training fails for two concrete numerical reasons:
- Rounding error: in $[2^{-3}, 2^{-2}]$,
FP16's fixed step spacing is $2^{-13}$, so adding gradient $2^{-14}$ to weight $2^{-3}$ rounds straight back to $2^{-3}$, silently dropping the update (andBF16's coarser 7-bit mantissa needsFP32master copies even more). - Gradient underflow: in late training, roughly 67% of activation gradients fall below $2^{-24}$ (the
FP16minimum normal positive limit) and vanish to zero—fixed by scaling loss by $S$ before backprop. BecauseBF16keepsFP32's 8-bit exponent ("get into the right range first, then worry about precision"), it avoids underflow and skips dynamic loss scaling.
Meanwhile, small sensitivity-critical tensors such as LayerNorm/BatchNorm parameters and the scalar loss remain in FP32 throughout.
Three low-level mechanics in Megatron's Float16OptimizerWithFloat16Params explain how distributed mixed precision works in code:
main_graddoes NOT mean "FP32 gradients": Under Megatron's local DDP (DDP_impl == "local",params_have_main_grad=True), parameters of the samedtypeshare a contiguousMemoryBuffer(accessed viatensor.main_gradinstead of fragmentedtensor.grad) soall_reduceruns in contiguous blocks. InFP16runs,main_gradis stillFP16and is only cast via.float()inside_copy_model_grads_to_main_grads().main_param = param.detach().clone().float(): Callingclone()alone leaves the master copy attached to autograd (requires_grad=True); callingdetach()alone shares storage withFP16weights. Chaining both isolates theFP32copy before copying TP attributes, preserving PP-shared Embedding flags (param.shared), and repointingoptimizer.param_groups.DynamicGradScaleruses cumulativehysteresis: Oninf/NaN, Megatron skips the optimizer update immediately, yet does not shrink $S$ on the first overflow. Instead it decrements cumulative_hysteresis_trackerand shrinksmax(_scale * backoff_factor, min_scale)only once it reaches<= 0, while scaling up bygrowth_factoraftergrowth_intervalconsecutive clean steps.
# Isolated FP32 master weight + cumulative hysteresis dynamic loss scaling [11]
main_param = param.detach().clone().float()
mpu.copy_tensor_model_parallel_attributes(main_param, param)
# DynamicGradScaler.update(found_inf): cumulative hysteresis backoff, consecutive growth
if found_inf:
self._growth_tracker = 0
self._hysteresis_tracker -= 1
if self._hysteresis_tracker <= 0:
self._scale = torch.max(self._scale * self.backoff_factor, self.min_scale)
else:
self._growth_tracker += 1
if self._growth_tracker == self.growth_interval:
self._growth_tracker = 0
self._hysteresis_tracker = self.hysteresis
self._scale = self._scale * self.growth_factor
Inside Float16OptimizerWithFloat16Params.step(), each iteration executes six steps in order:
- Copy & cast gradients (
_copy_model_grads_to_main_grads): Castmodel_param.main_grad(or.grad) intoFP32main_param.grad. - Unscale & global
found_infcheck (_unscale_main_grads_and_check_for_nan): Runtorch._amp_foreach_non_finite_check_and_unscale_, synchronizeself.found_infviadist.all_reduce(MAX)acrossmpu.get_model_parallel_group()($\text{TP} \times \text{PP}$), updategrad_scaler, and skip the step cluster-wide if any rank overflowed. - Global L2 clipping (
clip_grad_norm): Compute $\|g\|_2 = \sqrt{\sum_{r \in \text{TP}} \sum_{s \in \text{PP}} \|g_{r,s}\|_2^2}$ (excluding TP-replicated params such as LayerNorm so they are not counted TP times) and scale $g \leftarrow \frac{c}{\|g\|_2}g$ when $\|g\|_2 \gt c$. - Optional zero count: Log zero gradient counts via
count_zeros()if enabled. - Step optimizer (
self.optimizer.step()): UpdateFP32main_param(plus nativeFP32parameters). - Cast back (
_copy_main_params_to_model_params): Copy updatedFP32master weights back intoFP16/BF16model_paramvia Apex's single-kernel_multi_tensor_copy_this_to_that.
Sharding & Collectives
Device Mesh Notation
Once training spans tens or thousands of GPUs, verbal descriptions of which slice lives where break down quickly. Naming the cluster axes—such as a 32-GPU 2D $\text{Mesh}(X=8, Y=4)$—lets us write sharding directly onto tensor subscripts[1]:
- $A[I, J]$ (replicated): No subscript; every GPU holds the full tensor.
- $A[I_X, J]$ (sharded on one axis): Dimension $I$ is cut across physical axis $X$; omitting $Y$ means shards are replicated along $Y$.
- $A[I_X, J_Y]$ (2D sharded): Each card holds $\left(\frac{|I|}{|X|}, \frac{|J|}{|Y|}\right)$, shrinking memory by $1 / (|X|\cdot|Y|)$.
- $A[I_{XY}, J]$ (flattened multi-axis shard): Both $X$ and $Y$ combine into one $|X|\cdot|Y|$-way partition along $I$.
- $A[I_X, J_X]$ (invalid): Reusing the same hardware axis across two dimensions leaves only diagonal tiles.
- $C[I, K]\{U_X\}$ (unreduced partial sum): Every card along $X$ holds a full-shaped $[I, K]$ partial buffer whose true value is the elementwise sum across $X$.
import jax, jax.numpy as jnp
mesh = jax.make_mesh(axis_sizes=(8, 4), axis_names=('X', 'Y'))
P = lambda *spec: jax.NamedSharding(mesh, jax.sharding.PartitionSpec(*spec))
# Cut E along the 4-GPU Y axis; replicate across the 8 X-axis groups
A = jnp.zeros((2048, 8192), dtype=jnp.bfloat16, device=P('Y', None))
Consider a concrete BF16 matrix $A[2048, 8192]$ on $\text{Mesh}(X=8, Y=4)$. Writing $A[E_Y, F]$ cuts dimension $E$ across $Y$ alone ($4$ cards), so each GPU holds $2048 / 4 = 512$ rows ($512 \times 8192 \times 2\text{ B} = \mathbf{8\text{ MiB}}$) with $8$ identical replicas along $X$. Spreading rows across all 32 cards ($64$ rows each, $\mathbf{1\text{ MiB}}$) requires two subscripts: $A[E_{XY}, F]$. An easy sanity check confirms this: an AllGather_Y over 4 cards turns $512 \times 4$ cleanly back into $2048$ rows, whereas $64 \times 4 = 256$ falls short by the missing $8\times$ along $X$.
A[E_Y, F], A[E_XY, F], and A[E_Y, F_X] on a 32-GPU 8×4 mesh (click any cell to inspect).The Four Collectives
Here All refers to every GPU sharing a named mesh axis (e.g., the 4 ranks at a fixed $X$). Separating the four collectives comes down to two questions: Does data get summed elementwise? And does each card end with the whole tensor or a slice?
| Collective | Signature | Mechanics | Typical Usage |
|---|---|---|---|
AllGather |
$[A_X, B] \to [A, B]$ | Collect shards and concatenate along unsharded axis | FSDP / ZeRO-3 reconstructing weights before compute |
ReduceScatter |
$[A, B]\{U_X\} \to [A_X, B]$ | Sum elementwise, then scatter disjoint slices | Summing grads so each rank keeps its optimizer slice |
AllReduce |
$[A_X, B]\{U_Y\} \to [A_X, B]$ | Sum elementwise and replicate full result ($\text{RS} + \text{AG}$) | DDP gradient sync, Megatron Row-Parallel output |
All-to-All |
$[E_Y, F] \to [E, F_Y]$ | Split local data into $N$ chunks; send chunk $j$ to rank $j$ | MoE token routing, row-split to col-split reshard |
Running all four from the exact same 2-GPU initial state ($\text{GPU}_0 = [1, 2]$, $\text{GPU}_1 = [3, 4]$):
| Operation | GPU 0 Output | GPU 1 Output | Size Change |
|---|---|---|---|
| Initial input | $[1, 2]$ | $[3, 4]$ | Each GPU holds half ($1\times$) |
AllGather |
$[1, 2, 3, 4]$ | $[1, 2, 3, 4]$ | Concatenated without summing ($2\times$) |
ReduceScatter |
$[1+3] = [4]$ | $[2+4] = [6]$ | Summed then split ($0.5\times$) |
AllReduce |
$[4, 6]$ | $[4, 6]$ | ReduceScatter followed by AllGather ($1\times$ full) |
All-to-All |
$[1, 3]$ | $[2, 4]$ | Chunk 0 to GPU 0, chunk 1 to GPU 1 ($1\times$) |
Note the order: AllReduce is ReduceScatter + AllGather, not AllGather + Scatter. Plain Scatter only splits data without adding partial values like $1+3$ and $2+4$. Whenever the next consumer already operates on shards, stopping after ReduceScatter saves half the traffic.
AllReduce = ReduceScatter + AllGather.To compare AllGather against All-to-All, reuse the 4-GPU $Y$ group holding $A[E_Y, F]$ ($512 \times 8192$, $8\text{ MiB}$ per rank):
| Operation | Per-Rank Send Pattern | Output Shape | Local HBM |
|---|---|---|---|
| $\text{AllGather}_Y$ | Broadcast full 512 rows to all 3 peers | $[2048, 8192]$ | $32\text{ MiB}$ ($4\times$) |
| $\text{AllToAll}_Y$ | Split 8192 cols into 4 pieces; send $j$-th to rank $j$ | $[2048, 2048]$ | $8\text{ MiB}$ ($1\times$) |
Under AllToAll_Y, GPU 0 receives four $512 \times 2048$ blocks (all rows, first quarter of columns) and stacks them vertically—converting a row-sharded matrix into a column-sharded one with zero memory growth.
AllGather_Y (reconstructing full $32\text{ MiB}$ tensors) against AllToAll_Y (transposing row-shards to col-shards at constant $8\text{ MiB}$).One-line rule: Ask what the next op lacks:
- Full tensor from disjoint shards $\implies$
AllGather - Full replicated sum from partial buffers $\{U_X\}$ $\implies$
AllReduce - Sharded slice of a sum from $\{U_X\}$ $\implies$
ReduceScatter - Resharding across axes or routing MoE tokens $\implies$
All-to-All
Ring Cost & Duality
Let $V$ be total unsharded tensor bytes, $|X|$ be rank count along an axis, $W_{\text{ici}}$ be per-link bidirectional bandwidth, and $T_{\min}$ be per-hop latency. On a bidirectional ring, each GPU simultaneously sends $\frac{V}{|X|}$ chunks left and right over $\lfloor |X|/2 \rfloor$ hops; $|X|$ cancels out of the bandwidth term, and splitting across $N_{\text{axes}}$ orthogonal links scales bandwidth linearly:
$$T_{\text{AG / RS}} = \max\!\left(\frac{T_{\min} \cdot \sum_i |X_i|}{2},\; \frac{V}{W_{\text{ici}} \cdot N_{\text{axes}}}\right)$$For large tensors (bandwidth-bound), AllGather and ReduceScatter cost $\frac{V}{W_{\text{ici}}}$ regardless of group size $|X|$, while AllReduce runs both in sequence and costs $\frac{2V}{W_{\text{ici}}}$.
On the same bidirectional ring, All-to-All takes only one quarter the time of AllGather: AllGather sends every card's entire shard across all hops ($\propto D^2$), whereas All-to-All delivers each rank-specific $\frac{V}{D^2}$ piece only up to its destination distance ($1 + 2 + \dots + D/2$ in each direction), cutting total traffic by $4\times$. Generalized to a $D$-dimensional mesh $(A \times B \times \dots)$ of $N$ cards:
All-to-All is $4\times$ cheaper than AllGather.In autograd, collectives obey an exact transpose duality. Writing broadcast and reduce with an all-ones vector $\mathbf{u} \in \mathbb{R}^p$ and Kronecker product $\otimes$:
Since backprop through any linear map $\mathcal{A}$ multiplies by $\mathcal{A}^\top$, a forward AllGather differentiates automatically into a backward ReduceScatter—the exact algebraic backbone of FSDP and Sequence Parallelism.
Sharded Matmul
Every distributed linear layer evaluates $A[I, J] \cdot B[J, K] \to C[I, K]$ along contracting axis $J$, falling into one of four cases:
- Neither side cuts $J$: $A[I_X, J] \cdot B[J, K_Y] \to C[I_X, K_Y]$. Zero forward communication—covers Data Parallel forward and Megatron Column-Parallel layers.
- Only one side cuts $J$: $A[I, J_X] \cdot B[J, K]$. Usually preceded by
AllGather_X(FSDP), giving time $\max(\frac{2BDF}{C}, \frac{2DF}{W_{\text{ici}}})$, where communication hides behind compute once batch tokens $B > C / W_{\text{ici}}$. In real training activations are already sharded on the same axis $X$, so computing locally and callingAllReduceis blocked anyway. - Both sides cut $J$ on the same axis $X$: $A[I, J_X] \cdot_{\text{LOCAL}} B[J_X, K] \to C[I, K]\{U_X\}$. Ranks multiply locally to emit partial sums $\{U_X\}$, then finish with
AllReduce_X(Megatron Row-Parallel) orReduceScatterto preserve sharding (DP weight gradients / Sequence Parallelism). - Outer dimensions collide on $X$: $A[I_X, J] \cdot B[J, K_X]$. Invalid directly because rank $i$ only holds diagonal blocks $(I_i, K_i)$; call
AllGather_Xon the smaller factor first.
High-throughput kernels pipeline Cases 2 and 3 into a Collective Matmul, tiling Tensor Core MMA instructions alongside ring transfers of the next chunk.
AllGather, or post-AllReduce / ReduceScatter.Data Parallelism & ZeRO
DP and DDP
Pure Data Parallelism runs Case 1 forward (sharded batch $B_{\text{DP}}$) and Case 3 backward on weight gradients $dW = X[B_{\text{DP}}, D]^\top \cdot dY[B_{\text{DP}}, F]$, contracting along $B_{\text{DP}}$ to produce $\{U_{\text{DP}}\}$.
Under the classic Parameter Server DP model[8], $N$ compute Workers push local gradients to a single Server (often GPU 0 on a single node) and pull back averaged updates. Total cluster traffic is $N\Phi\text{ (Server)} + N\Phi\text{ (Workers)} = 2N\Phi$, so the Server scales linearly in $N$ and saturates its NIC across nodes (though multi-Server setups shard parameters across servers).
To keep Workers busy while the Server syncs, PS frameworks also support asynchronous bounded-delay updates—letting a Worker advance iter 11 on stale weights while iter 10 syncs (e.g., enforcing iter 10–11 applied before iter 12 under 1-bounded delay), trading slower SGD convergence for hardware utilization.
DDP eliminates the central Server using Ring-AllReduce: each GPU slices its gradients (not input batch) into $N$ chunks of $\Phi/N$ and runs $N-1$ hops of ReduceScatter ($(N-1)\frac{\Phi}{N}$ send/GPU) followed by $N-1$ hops of AllGather ($(N-1)\frac{\Phi}{N}$ send/GPU), fixing per-GPU send volume at a rank-independent $2\frac{N-1}{N}\Phi \approx \mathbf{2\Phi}$.
In PyTorch DDP, one process per GPU avoids Python's GIL while a C++ Reducer packs parameters in reverse topological order into ~25 MB buckets so NCCL overlaps asynchronous ring reductions behind ongoing backward passes.
ZeRO and FSDP
DDP balances bandwidth yet keeps $N$ redundant $16\Phi$ copies of model states and repeats identical Adam updates on every card. DeepSpeed ZeRO[4][9] (and PyTorch FSDP) shards optimizer states ($P_{os}$), gradients ($P_g$), and parameters ($P_p$) across $N_d$ DP ranks ("consume and discard, re-gather when needed"):
Tracking our 2-GPU gradients $g_0=[1,2]$ and $g_1=[3,4]$: DDP runs AllReduce(g) so both GPUs hold $[4,6]$ and update all weights; ZeRO-3 / FSDP permanently stores only $W_0, \text{OS}_0$ on GPU 0, runs AllGather(W) right before computing each layer (and frees peer shards immediately after), then executes ReduceScatter(g) so GPU 0 receives only $[4]$ to update $W_0$ while GPU 1 gets $[6]$ for $W_1$:
| Strategy | Sharded | Static HBM | 7.5B (N=64) | Comm / Rank |
|---|---|---|---|---|
DDP |
None | $16\Phi$ | 120 GB | $2\Phi$ ($1\times$) |
ZeRO-1 |
Optimizer States $P_{os}$ | $4\Phi + \frac{12\Phi}{N_d}$ | 31.4 GB | $2\Phi$ (paper $3\Phi$) |
ZeRO-2 |
$P_{os}$ + Gradients $P_g$ | $2\Phi + \frac{14\Phi}{N_d}$ | 16.6 GB | $2\Phi$ ($1\times$) |
ZeRO-3 / FSDP |
$P_{os}$ + $P_g$ + Params $P_p$ | $\frac{16\Phi}{N_d}$ | 1.9 GB | $3\Phi$ ($1.5\times$) |
ZeRO-1 $3\Phi$ vs. $2\Phi$ traffic & gradient accumulation: conceptually the ZeRO paper counts Stage-1 traffic as full-grad AllReduce ($2\Phi$) + AllGather(W) ($\Phi$) = $3\Phi$, whereas modern implementations swap AllReduce(G) for ReduceScatter(G) ($\Phi$) to stay at $2\Phi$. Still, keeping full gradients in Stage 1 saves $m-1$ sync steps under $m$-step gradient accumulation compared with Stage 2.
PyTorch Memory Snapshots on 2 GPUs reveal two more ZeRO-1 implementation details:
- In-place
FP16weight refresh: updated weights write directly viafp16._copy_(fp32), reusing persistent GPU buffers. - Lazy Adam state allocation: momentum $m, v$ tensors are allocated only on the first clean step without
NaN/infoverflow.
ZeRO is best summarized as "model-parallel in form, data-parallel in substance": even Stage 3 processes disjoint micro-batches with transiently gathered full layer weights (freeing peer shards immediately after each layer).
Alongside model states, ZeRO-R targets residual memory from three angles:
- $P_a$ (partitioned activations): shards activations replicated across TP ranks (pure DP activations are already batch-split).
- $C_B$ (constant-size buffers): batches small cross-GPU slices up to a fixed buffer cap to keep interconnect bandwidth saturated.
- $M_D$ (memory defragmentation): consolidates fragmented HBM left behind by mixed short- and long-lived tensors.
When HBM remains tight, ZeRO-Offload / ZeRO-Infinity keeps compute-heavy forward/backward passes on GPUs while offloading large, low-FLOP FP32 master weights and optimizer steps to CPU RAM.
Tensor Parallelism
MLP, Attention & Vocab
ZeRO reduces static state, not intra-layer activations. Tensor Parallelism (TP)[2][10] splits each linear matmul $Y = XW$ ($X:[b,s,h], W:[h,h']$) in one of two ways:
- Row-split (along $h$): forces a matching column-split on $X$, sums partial outputs via $g$ in forward, and replicates/concatenates input gradients via $f$ in backward.
- Column-split (along $h'$): replicates $X$, concatenates output slices $[Y_1,Y_2]$, and relies on $f$ in backward to
AllReduce$\partial\mathcal{L}/\partial X|_1 + \partial\mathcal{L}/\partial X|_2$.
Megatron-LM pairs Column-Parallel then Row-Parallel linears so nonlinearities and softmax run with zero communication:
- MLP: Column-split $A = [A_0, A_1]$ and row-split $B = \begin{bmatrix} B_0 \\ B_1 \end{bmatrix}$. Because $\text{GeLU}(Y_1+Y_2) \neq \text{GeLU}(Y_1)+\text{GeLU}(Y_2)$ whereas $\text{GeLU}([Y_1,Y_2]) = [\text{GeLU}(Y_1),\text{GeLU}(Y_2)]$, column-splitting $A$ avoids syncing before GeLU and uses only 1 forward + 1 backward
AllReduce($4bsh$ total): $$Z = \text{AllReduce}\!\left(\sum_i \text{GeLU}(X A_i)\, B_i\right)$$ - Multi-Head Attention: Split $W^Q, W^K, W^V$ along
k_dimby heads ($\text{num\_heads} \bmod \text{TP} = 0$) and row-split $W^O$, also costing $4bsh$—giving 4AllReduces ($8bsh$) per Transformer layer. Adding Sequence Parallelism (SP)[3] splits these intoAllGatherandReduceScatterat identical traffic while eliminating TP-replicated LayerNorm/Dropout activations. - Vocab Embedding &
VocabParallelCrossEntropy: Position embeddings $[s_{\max}, h]$ stay replicated while word embeddings $[V, h]$ split along $V$. For token IDs[0,212,7,9]across $[0,150)$ and $[150,299)$, each GPU masks out-of-range IDs to zero ([ok,0,ok,ok]+[0,ok,0,0]) and runs oneAllReduce. When $\text{PP} \gt 1$, input and output tie the same word embedding, so Megatron must run an extraAllReduceacross the first and last stages before stepping. On output logits $[b,s,V/\text{TP}]$, exact Log-Sum-Exp normalization avoids gathering $[b,s,V]$ by syncing only three lightweight $[b,s]$ scalars: $$\begin{aligned} m_{\text{global}} &= \text{AllReduce}_{\max}\!\left(\max_{j \in V_i} z_{i,j}\right), \qquad z'_{i,j} = z_{i,j} - m_{\text{global}} \\ S_{\text{global}} &= \text{AllReduce}_{\text{sum}}\!\left(\sum_{j \in V_i} e^{z'_{i,j}}\right), \qquad z'_{\text{target}} = \text{AllReduce}_{\text{sum}}\!\left(\mathbb{I}[y \in V_i]\, z'_{i,y}\right) \\ \mathcal{L} &= \log S_{\text{global}} - z'_{\text{target}} \end{aligned}$$
Why keep TP intra-node and push DP inter-node? Two factors drive the topology choice:
- Per-layer volume: TP sends $8bsh$ vs. DP's $2h^2$—canceling $h$ compares $[b\cdot s]$ against $[h]$ (both $\sim 10^5$, with TP typically larger at long sequence lengths).
- The decisive bottleneck—backward blocking: DP launches gradient
AllReduceasynchronously while earlier layers keep differentiating, whereas TP stalls every single layer untilAllReducereturns $\partial\mathcal{L}/\partial X$.
Across 32× DGX-2H nodes (512× V100-32GB) training GPT-2 (1.2B→8.3B), pure TP scales at 100%→95%→82%→77% from 1 to 8 GPUs (capping at 8/16 GPUs per node due to activation HBM), while pairing intra-node $\text{TP}=8$ with inter-node $\text{DP}=64$ reaches 15,100 TFLOPs out of $39 \times 512 = 19{,}968\text{ TFLOPs}$ ideal—76% weak scaling efficiency.
Megatron Internals
Three low-level mechanics inside Megatron's mpu (parallel_state) module keep 3D parallelism fast and numerically identical to single-GPU training:
1. 3D process group layout (initialize_model_parallel): Megatron orders global ranks from innermost TP → middle DP → outermost PP. With 16 GPUs across two 8-GPU nodes ($\text{TP}=2, \text{PP}=2, \text{DP}=4$), it constructs four orthogonal process groups:
- TP groups (stride $1$, eight 2-GPU pairs):
[0,1], [2,3], ..., [14,15]—adjacent ranks stay inside the same intra-node NVLink switch. - DP groups (stride $\text{TP}=2$, four 4-GPU groups):
[0,2,4,6]&[1,3,5,7]on Stage 0 and[8,10,12,14]&[9,11,13,15]on Stage 1, syncing matching weight shards across replicas. - PP groups (stride $\text{world}/\text{PP}=8$, eight 2-stage pipelines):
[0,8], [1,9], ..., [7,15], chaining nodes across pipeline stages. - Model-parallel groups $\text{MP} = \text{TP} \times \text{PP}$ (four 4-GPU replicas):
[0,1,8,9], [2,3,10,11], [4,5,12,13], [6,7,14,15], used for the globalfound_infconsensus check and global L2clip_grad_norm.
initialize_model_parallel for 16 GPUs ($\text{TP}=2, \text{PP}=2, \text{DP}=4$).In code, each rank computes the exact same sorted lists and calls torch.distributed.new_group(ranks) in lockstep—every process must call new_group in identical global order even when rank not in ranks so collective group creation never deadlocks:
def initialize_model_parallel(tp_size: int = 2, pp_size: int = 2):
world_size = dist.get_world_size() # e.g., 16 = TP(2) * PP(2) * DP(4)
rank = dist.get_rank()
dp_size = world_size // (tp_size * pp_size)
num_tp_groups = world_size // tp_size # 8 TP groups
num_pp_groups = world_size // pp_size # 8 PP pipelines
# 1. DP groups (stride = tp_size): [0,2,4,6], [1,3,5,7], ...
for i in range(pp_size):
start = i * num_pp_groups
end = (i + 1) * num_pp_groups
for j in range(tp_size):
ranks = list(range(start + j, end, tp_size))
group = dist.new_group(ranks)
if rank in ranks:
_DATA_PARALLEL_GROUP = group
# 2. Tensor Parallel groups (contiguous, stride = 1): [0,1], [2,3], ..., [14,15]
for i in range(num_tp_groups):
ranks = list(range(i * tp_size, (i + 1) * tp_size))
group = dist.new_group(ranks)
if rank in ranks:
_TENSOR_MODEL_PARALLEL_GROUP = group
# 3. Pipeline Parallel groups (stride = num_pp_groups = 8): [0,8], [1,9], ..., [7,15]
for i in range(num_pp_groups):
ranks = list(range(i, world_size, num_pp_groups))
group = dist.new_group(ranks)
if rank in ranks:
_PIPELINE_MODEL_PARALLEL_GROUP = group
# 4. Full Model Parallel groups (TP x PP): [0,1,8,9], [2,3,10,11], [4,5,12,13], [6,7,14,15]
for i in range(dp_size):
ranks = [i * tp_size + p * num_pp_groups + t
for p in range(pp_size) for t in range(tp_size)]
group = dist.new_group(ranks)
if rank in ranks:
_MODEL_PARALLEL_GROUP = group
2. Conjugate autograd ops f/g and linear wiring: ColumnParallelLinear applies f at its input (keeping gather_output=False so GeLU runs on sharded activations), and RowParallelLinear (input_is_parallel=True) closes with g. Because forward sends replicated $X$ into all $A_i$, chain-rule backprop $\frac{\partial\mathcal{L}}{\partial X} = \sum_i \frac{\partial\mathcal{L}}{\partial Y_i}A_i^\top$ requires summing gradients inside f.backward:
class _CopyToModelParallelRegion(torch.autograd.Function):
"""Op f: Identity in forward, AllReduce in backward (ReduceScatter under SP)."""
@staticmethod
def forward(ctx, input_): return input_
@staticmethod
def backward(ctx, grad_output): return _reduce(grad_output)
class _ReduceFromModelParallelRegion(torch.autograd.Function):
"""Op g: AllReduce in forward (ReduceScatter under SP), Identity in backward."""
@staticmethod
def forward(ctx, input_): return _reduce(input_)
@staticmethod
def backward(ctx, grad_output): return grad_output
# ParallelMLP flow (zero comm across GeLU):
# x -> f -> ColumnParallelLinear(gather_output=False) -> GeLU
# -> RowParallelLinear(input_is_parallel=True) -> g -> out
f / g operators, with a toggle for Sequence Parallelism (TP + SP).3. CudaRNGStatesTracker: Dropout masks inside TP-sharded activations use a distinct rank-seeded state (model-parallel-rng, seeded with base_seed + 2718 + tp_rank), whereas Dropout on replicated residual paths exits the context so every TP rank shares the identical base mask:
with get_cuda_rng_tracker().fork('model-parallel-rng'):
# Sharded activations (e.g., inside attention): independent masks per tp_rank
attention_probs = self.attention_dropout(attention_probs)
# Restored base_seed on replicated residual path: identical mask across TP ranks
hidden_states = self.hidden_dropout(output)
Because TP's blocking collectives sit directly on the layer critical path and cannot hide behind asynchronous buckets, practical clusters keep $\text{TP} \le 8$ within a single NVLink node.
Pipeline Parallelism
GPipe and 1F1B
When a model outgrows 8 GPUs, pushing TP over cross-node Ethernet/InfiniBand tanks utilization. Pipeline Parallelism (PP) slices the model vertically into $P$ stages connected only by lightweight P2P send/recv transfers of boundary activations.
Feeding one unsplit batch leaves $P-1$ stages idle ($\frac{P-1}{P} = 75\%$ bubble at $P=4$). GPipe[5] splits each mini-batch into $M$ micro-batches; assuming equal forward and backward step times, idle bubble overhead shrinks to:
$$r_{\text{GPipe}} = \frac{P - 1}{M + P - 1}$$Taking $M \ge 4P$ drops bubble time below $16\%$.
Stashing full intermediate activations for all $M$ micro-batches in flight would still blow HBM, so GPipe pairs scheduling with activation checkpointing (re-materialization): forward stores only stage boundary tensors and recomputes internal layers on the backward pass, trading ~33% extra FLOPs for peak memory $\mathcal{O}\!\left(N + \frac{L}{P}\cdot\frac{N}{M}\right)$.
Even with recomputation, GPipe holds all $M$ micro-batches at Stage 0 until every forward pass completes, coupling memory to $\mathcal{O}(M)$. Pipelining schedules break that coupling:
- 1F1B: alternates one forward step and one backward step as soon as the pipeline fills, locking peak buffered micro-batches at $\mathcal{O}(P)$ while keeping the same bubble ratio.
- Interleaved 1F1B: assigns $v$ non-contiguous layer chunks per GPU to shrink the bubble to $\frac{P-1}{vM + P - 1}$ in exchange for $v\times$ tiny P2P transfers.
Because first and last stages also carry Embeddings and output heads, practical setups place fewer Transformer layers at both ends to balance stage compute and memory.
MoE & Expert Parallelism
Routing & Load Balance
Mixture-of-Experts (MoE) decouples parameter count from per-token FLOPs by replacing each FFN with $E$ expert networks and dynamically routing every token via $P = \text{Softmax}(x W_g)$ to its top-$k$ experts.
To keep static tensor shapes on GPUs, each expert caps its buffer at a fixed Expert Capacity[6]:
With capacity_factor typically in $1.0\text{–}1.5$, overflow tokens bypass the expert via the residual connection (dropped), while underfilled slots are zero-padded.
Because unregularized routers collapse onto one or two popular experts, MoE adds a differentiable load-balancing loss ($\alpha$ scaled by $E$ so magnitude stays invariant to expert count):
$$\mathcal{L}_{\text{aux}} = \alpha \cdot E \sum_{i=1}^{E} f_i \cdot P_i$$Here $f_i$ is the actual fraction of tokens dispatched to expert $i$ and $P_i$ is mean router probability; by Cauchy-Schwarz this attains its unique global minimum $\alpha$ iff $f_i = P_i = 1/E$.
All-to-All & EP Groups
Whereas ZeRO-3 keeps tokens stationary and moves weights via AllGather, Expert Parallelism (EP)[7] pins massive expert weights on fixed GPUs and ships lightweight tokens over two All-to-All passes—capitalizing on its $4\times$ bandwidth advantage on a ring.
On two GPUs where GPU 0 hosts $E_0$ and holds $[a, b]$, and GPU 1 hosts $E_1$ and holds $[c, d]$ (routing $a,c \to E_0$ and $b,d \to E_1$): Pass 1 (Dispatch) exchanges $b$ and $c$ so each GPU computes its local experts; Pass 2 (Combine) returns $[a', b']$ and $[c', d']$ to their origin ranks for the next Attention block:
class _AllToAll(torch.autograd.Function):
@staticmethod
def forward(ctx, group, x):
ctx.group = group
out = torch.empty_like(x)
dist.all_to_all_single(out, x, group=group)
return out
@staticmethod
def backward(ctx, *grad_out):
# All-to-All is self-adjoint
return (None, _AllToAll.apply(ctx.group, *grad_out))
dispatched = torch.einsum("sec,sm->ecm", dispatch_mask, x)
dispatched = _AllToAll.apply(ep_group, dispatched) # Pass 1: dispatch
expert_out = experts(dispatched)
combined = _AllToAll.apply(ep_group, expert_out) # Pass 2: combine
output = torch.einsum("sec,ecm->sm", combine_weights, combined)
All-to-All token routing cycle and a skewed 4-GPU CF=1.0 example showing capacity drops.In DeepSpeed-Megatron, shared parameters (Attention, Embeddings, LayerNorm, Router) replicate across all DP ranks while experts shard along $\text{EP}$. With 8 DP GPUs, 4 experts, and $\text{EP}=4$, the cluster forms $2$ full expert replicas—splitting DP cleanly as $\text{DP}(8) = \text{EP}(4) \times \text{EDP}(2)$:
EP groups({0,1,2,3},{4,5,6,7}): Route tokens via twoAll-to-Allpasses.EDP groups({0,4}, {1,5}, {2,6}, {3,7}): Sync gradients viaAllReducestrictly across identical expert replicas—never averaging $E_0$ with $E_1$.- Global
DP group({0..7}): Sync all shared non-expert parameters across every rank.
Inside each expert FFN, intra-node TP can still be nested normally, maintaining $\text{world\_size} = \text{PP} \times \text{DP} \times \text{TP}$ with $\text{DP} = \text{EP} \times \text{EDP}$.
DP(8) = EP(4) × EDP(2)) separating token dispatch, expert replica sync, and shared-weight DP sync.4D Hybrid Parallelism
Putting every strategy side by side makes cluster topology design mechanical:
| Strategy | Sharded Axis | Communication | Memory Saved | Target Link |
|---|---|---|---|---|
DDP |
Batch $B_{\text{DP}}$ | Async bucketed AllReduce |
Batch activations $/ \text{DP}$ | Outer IB / RoCE |
ZeRO-1 / 2 |
+$P_{os}$ / $P_g$ | Grad RS + updated weight AG |
OS/G $/ \text{DP}$ at $2\Psi$ total | Outer IB / RoCE |
ZeRO-3 / FSDP |
+$P_p$ Weights | Layerwise AG(W) + RS(g), total $3\Psi$ |
Static state $16\Phi / \text{DP}$ | NVLink / fast IB |
TP / TP+SP |
Col / Row (+$S_{\text{TP}}$) | 2 blocking syncs fwd & bwd | Weights & all acts $/ \text{TP}$ | Intra-node NVLink $(\le 8)$ |
PP (1F1B) |
Layers ($P$ stages) | Light P2P send/recv |
States $/ P$, acts $\mathcal{O}(P)$ | Cross-node IB |
EP (MoE) |
Experts ($E_{\text{EP}}$) | 2 All-to-All per MoE layer |
Expert states $/ \text{EP}$ | NVLink / full-bisection IB |
On 64 × H100-80GB (8 nodes of 8 GPUs), map the heaviest communication onto the fastest fabric first:
- 70B dense ($16\Phi \approx 1.12\text{ TB}$): Lock $\text{TP}=8$ with Sequence Parallelism inside each 8-GPU NVLink node ($140\text{ GB}$/rank static); add cross-node $\text{PP}=2$ with 1F1B ($M \ge 8$, ~11% bubble) to cut static state to $70\text{ GB}$; use the remaining $\text{DP}=64/(8 \times 2)=4$ lanes with zero-overhead ZeRO-1, bringing per-GPU static HBM to $\frac{4\Phi}{16} + \frac{12\Phi}{64} = 17.5 + 13.1 = \mathbf{30.6\text{ GB}}$ and leaving ~50 GB free for long-context activations.
- 64-GPU MoE: Pair intra-node $\text{TP}=4$ with $\text{PP}=2$, then split $\text{DP}=8$ into $\text{EP}=4, \text{EDP}=2$ so 4-GPU groups exchange tokens via
All-to-All, 2-way EDP syncs matching experts, and global DP + ZeRO-1 updates shared Attention and Router weights.
Citation
If you reference these notes or interactive animations, please cite:
Chaoran Feng (2026). "Distributed LLM Training: Sharding, Collectives, and Parallelism". Chaoran Feng's Blog. https://www.falcary.com/blogs/distributed-training/
@article{feng2026distributed,
title = {Distributed LLM Training: Sharding, Collectives, and Parallelism},
author = {Feng, Chaoran},
year = {2026},
month = {October},
url = {https://www.falcary.com/blogs/distributed-training/}
}
References
- How to Scale Your Model, Part 3: Sharded Matrices and How to Multiply Them. Austin, Bradbury, Blake, Chou, Douglas, Frostig, Levskaya, Pope, et al., Google DeepMind, 2025.
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. Shoeybi et al., NVIDIA, 2019 (arXiv:1909.08053).
- Reducing Activation Recomputation in Large Transformer Models. Korthikanti et al., NVIDIA, 2022 (arXiv:2205.05198).
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. Rajbhandari et al., Microsoft DeepSpeed, SC 2020 (arXiv:1910.02054).
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. Huang et al., Google Brain, NeurIPS 2019 (arXiv:1811.06965).
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. Lepikhin et al., Google Brain, ICLR 2021 (arXiv:2006.16668).
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. Rajbhandari et al., ICML 2022 (arXiv:2201.05596).
- 图解大模型训练之:数据并行上篇(DP, DDP 与 ZeRO),猛猿(公众号:大猿搬砖简记),知乎。
- 图解大模型训练之:数据并行下篇(DeepSpeed ZeRO,零冗余优化),猛猿(公众号:大猿搬砖简记),知乎。
- 图解大模型训练之:张量模型并行(TP),Megatron-LM,猛猿(公众号:大猿搬砖简记),知乎。
- 图解大模型训练系列之:Megatron 源码解读 3,分布式混合精度训练,猛猿(公众号:大猿搬砖简记),知乎。