Knowledge distillation

A technique for training a smaller, cheaper "student" model to reproduce the behaviour of a larger, more capable "teacher" model, so that most of the teacher's performance can be preserved at a fraction of the computing cost. Distillation is one of the main ways smaller language models are produced from larger foundation models. (See also: Small language model, Foundation model)