What is knowledge distillation in NLP models?

Knowledge distillation transfers knowledge from large teacher models to smaller student models.

A large teacher model trains a smaller student model by providing soft probability distributions instead of hard labels. This helps compress models while retaining performance.