How do you reduce latency in LLM systems?

Latency is reduced using caching, smaller models, quantization, and optimized inference pipelines.

Latency reduction techniques include KV caching, speculative decoding, model distillation, and prompt shortening. Infrastructure optimizations like GPU batching and async pipelines also help.