Learning from Teaching Regularization: Generalizable Correlations Should
  be Easy to Imitate
Can Jin Tong Che Hongwu Peng Yiyuan Li Marco Pavone
Generalization remains a central challenge in machine learning. In this work,
we propose Learning from Teaching (LoT), a novel regularization technique for
deep neural networks to enhance generalization. Inspired by the human ability
to capture concise and abstract patterns, we hypothesize that generalizable
correlations are expected to be easier to teach. LoT operationalizes this
concept to improve the generalization of the main model with auxiliary student
learners. The student learners are trained by the main model and improve the
main model to capture more generalizable and teachable correlations by
providing feedback. Our experimental results across several domains, including
Computer Vision, Natural Language Processing, and Reinforcement Learning,
demonstrate that the introduction of LoT brings significant benefits compared
to merely training models on the original training data. It suggests the
effectiveness of LoT in identifying generalizable information without falling
into the swamp of complex patterns in data, making LoT a valuable addition to
the current machine learning frameworks.