What is model parallelism vs pipeline parallelism in distributed training?

Model parallelism splits model layers across devices, while pipeline parallelism splits computation into staged execution pipelines.

Model parallelism divides a single model across GPUs when it is too large to fit in memory. Pipeline parallelism divides the model into sequential stages, where each GPU processes a different stage in a pipeline fashion. Pipeline parallelism improves utilization but introduces pipeline bubbles (idle time). Both approaches are often combined with data parallelism in large-scale LLM training.