What is knowledge distillation in Computer Vision models?

Knowledge distillation transfers knowledge from a large teacher model to a smaller student model.

A large pretrained teacher model generates soft probabilities, which a smaller student model learns to mimic. This helps compress models while retaining performance. The loss combines hard labels and soft teacher outputs using temperature scaling.