What is model quantization in LLMs?

Quantization reduces numerical precision of model weights to improve inference speed and reduce memory usage.

LLMs are computationally expensive because model weights are typically stored in high precision formats like FP32 or FP16. Quantization compresses these weights into lower precision representations such as INT8 or INT4.

Benefits include:

The trade-off is potential quality degradation if precision loss becomes excessive.

Modern techniques such as GPTQ, AWQ, and QLoRA minimize quality degradation while maximizing efficiency.