What is the difference between static and dynamic batching in inference systems?

Static batching groups fixed-size inputs; dynamic batching groups requests at runtime.

Static batching requires pre-formed batches with fixed sizes, improving throughput but increasing latency. Dynamic batching collects incoming requests and batches them on the fly, balancing latency and throughput in serving systems.