Towards a theory of model distillation
Enric Boix-Adsera
Distillation is the task of replacing a complicated machine learning model
with a simpler model that approximates the original [BCNM06,HVD15]. Despite
many practical applications, basic questions about the extent to which models
can be distilled, and the runtime and amount of data needed to distill, remain
largely open.
  To study these questions, we initiate a general theory of distillation,
defining PAC-distillation in an analogous way to PAC-learning [Val84]. As
applications of this theory: (1) we propose new algorithms to extract the
knowledge stored in the trained weights of neural networks -- we show how to
efficiently distill neural networks into succinct, explicit decision tree
representations when possible by using the ``linear representation
hypothesis''; and (2) we prove that distillation can be much cheaper than
learning from scratch, and make progress on characterizing its complexity.