What is inference batching and dynamic batching?

Inference batching groups requests to improve throughput and GPU utilization.

Static batching processes fixed-size groups, while dynamic batching aggregates requests in real time based on arrival rate. It improves throughput but introduces latency trade-offs. Widely used in GPU inference servers like TensorRT or Triton.