Distillation

Distillation (Hinton et al., 2015) trains a student model on the teacher's soft outputs (probability distributions) rather than just correct labels. The student learns not just what to predict but how confident to be, capturing more nuance than label-only training. Production AI uses distillation to ship smaller, faster variants: Claude Haiku is distilled from Claude Sonnet/Opus recipes; GPT-4o Mini is distilled from GPT-4o; Gemini Flash from Gemini Pro.

Teacher-student distillation: the student minimizes KL divergence between its output distribution and the teacher's on a training set. Temperature softens the teacher distribution · higher temperatures surface information about non-top tokens. Distillation targets: logits (most common), hidden states (feature matching), attention maps (relation-matching), or generated sequences (sequence-level). For large-scale distillation of frontier models, sequence-level works best · train the student on teacher-generated outputs as if they were ground truth. Cost: producing a high-quality distillation corpus is expensive (you pay to run the teacher at scale), but once done, student training is cheap.

KL-divergence loss: L = T² · KL(softmax(z_t/T) || softmax(z_s/T)) where T is temperature and z_t, z_s are teacher/student logits. Large T (4-10) reveals dark knowledge · probability mass in non-top tokens. Modern pipelines combine distillation with SFT on high-quality human data: student trains on both teacher outputs and human-labeled examples. For MoE distillation, the student is often a dense model (e.g., DeepSeek R1-Distill variants) · experts don't transfer cleanly to smaller architectures. Quality recovery: a 10× smaller student typically matches 80-95% of teacher benchmark performance depending on task complexity.