How does batching strategy impact latency and throughput in ChatGPT serving architecture?

Batching increases throughput by processing multiple requests together but can increase latency for individual requests.

In ChatGPT serving systems, batching combines multiple user requests into a single forward pass through the model. This improves GPU utilization and throughput because matrix operations are more efficient when parallelized.

However, batching introduces a tradeoff: requests must wait for a batch to fill, increasing latency for individual users. Advanced systems use dynamic batching and continuous batching to balance this tradeoff by grouping requests based on arrival time and token generation stage.…