What is Knowledge Distillation in Deep Learning?

Knowledge Distillation transfers knowledge from a large teacher model to a smaller student model for efficient deployment.

Modern deep learning models achieve high accuracy but are often too large for production systems. Knowledge Distillation compresses these models by teaching smaller networks to imitate larger ones.

Architecture:

Instead of training only on hard labels, the student learns from:

Soft targets contain richer information.

Example: Hard label: Cat = 1 Dog = 0…