AI distillation

AI Distillation: Teaching Smaller Models Hugging Face’s latest live tutorial dives deep into model distillation, a technique where a smaller student model learns from a larger teacher model. The session covers four key axes—signal, data source, timing, and teacher identity—and explores methods like off-policy (training on teacher-generated data), on-policy (scoring student outputs live), and self-distillation … Read more