How does distributed model parallelism enable ChatGPT-scale transformer inference across GPUs?

Distributed model parallelism splits a large transformer across multiple GPUs so each GPU handles part of the model computation.

ChatGPT-scale models are too large to fit on a single GPU, requiring model parallelism techniques. Tensor parallelism splits matrix multiplications across GPUs, while pipeline parallelism splits layers across devices.

During inference, activations flow between GPUs, requiring high-bandwidth interconnects (e.g., NVLink or InfiniBand). Synchronization overhead becomes a key bottleneck.

This architecture allows extremely large models to run but introduces tradeoffs in latency due to communication between devices.