What is Model Distillation

Definition

Model distillation is a technique for compressing a large model into a smaller one by training the compact 'student' model to mimic the outputs of the larger 'teacher', retaining much of the performance while greatly reducing size, cost, and latency.

« Back to Glossary Index

Produces small, fast models that retain much of a large model’s quality
Cuts inference cost and latency for production deployment
Enables capable models on resource-constrained or edge environments
Preserves teacher behaviour without its full computational expense

Real World Example

A company distills its large language model into a smaller student model for its customer-support feature, cutting per-query cost and response time while keeping answer quality close to the original.

FAQs

How does distillation work?

A smaller student model is trained to reproduce the outputs or internal signals of a larger teacher model, transferring its capability.

Why distill a model?

To get faster, cheaper inference and smaller deployment footprints while retaining most of the larger model's performance.

What is the trade-off?

The distilled model usually loses some accuracy or capability compared with the teacher, traded for efficiency.