What is pipeline parallelism and how does it differ from tensor parallelism?

Pipeline parallelism splits model layers across devices; tensor parallelism splits computation within layers.

Pipeline parallelism assigns sequential layers to different GPUs and processes micro-batches in stages. Tensor parallelism splits matrix operations across GPUs within a single layer. Pipeline improves model depth scaling, tensor improves width scaling.