What is model quantization in NLP?

Quantization reduces model precision to make inference faster and smaller.

Quantization converts floating-point weights into lower precision formats like INT8 or FP16. This reduces memory footprint and increases inference speed, often with minimal loss in accuracy when done carefully.