What are trade-offs between model size and inference latency in NLP systems?

Larger models improve accuracy but increase latency and compute cost.

Inference latency scales with parameter count and sequence length. Larger models require more memory bandwidth and compute cycles. Techniques like distillation, quantization, and MoE help balance accuracy and efficiency trade-offs in production.