Curriculum Learning with Adam: The Devil Is in the Wrong Details
Lucas Weber Jaap Jumelet Paul Michel Elia Bruni Dieuwke Hupkes
Curriculum learning (CL) posits that machine learning models -- similar to
humans -- may learn more efficiently from data that match their current
learning progress. However, CL methods are still poorly understood and, in
particular for natural language processing (NLP), have achieved only limited
success. In this paper, we explore why. Starting from an attempt to replicate
and extend a number of recent curriculum methods, we find that their results
are surprisingly brittle when applied to NLP. A deep dive into the
(in)effectiveness of the curricula in some scenarios shows us why: when
curricula are employed in combination with the popular Adam optimisation
algorithm, they oftentimes learn to adapt to suboptimally chosen optimisation
parameters for this algorithm. We present a number of different case studies
with different common hand-crafted and automated CL approaches to illustrate
this phenomenon, and we find that none of them outperforms optimisation with
only Adam with well-chosen hyperparameters. As such, our results contribute to
understanding why CL methods work, but at the same time urge caution when
claiming positive results.