How does token-level parallelism differ from sequence-level parallelism in ChatGPT inference?

Token-level parallelism processes multiple tokens within computation steps, while sequence-level parallelism processes multiple sequences concurrently.

In ChatGPT inference, parallelism is essential for scaling performance. Sequence-level parallelism batches multiple user requests together, allowing simultaneous processing across GPU cores.

Token-level parallelism, however, splits computation of a single sequence across hardware units (e.g., splitting attention or matrix operations). This is more complex but allows large models to fit across multiple GPUs.

Both approaches are often combined: sequence-level batching improves throughput, while token-level parallelism enables model scalability.