Explain data parallelism versus model parallelism for large-scale training on SageMaker

Data parallelism replicates the model on every GPU and shards the batch, synchronising gradients each step; model parallelism splits the model itself when it cannot fit in one GPU's memory. Modern practice combines them, usually via sharded data parallelism such as FSDP or SMP.

The decision is driven by one question: does the model plus its optimizer state fit on a single GPU?

Memory for mixed-precision Adam is roughly params × (2 bytes fp16 weights + 4 fp32 master + 4 momentum + 4 variance) ≈ 14 bytes/param, plus activations. A 7B model therefore needs ~98 GB of state before activations — impossible on an 80 GB A100, which is why pure data parallelism breaks down well below the model sizes people assume.…