ChapGTP, ILLC's Attempt at Raising a BabyLM: Improving Data Efficiency
  by Automatic Task Formation
Jaap Jumelet Michael Hanna Marianne de Heer Kloots Anna Langedijk Charlotte Pouw Oskar van der Wal
We present the submission of the ILLC at the University of Amsterdam to the
BabyLM challenge (Warstadt et al., 2023), in the strict-small track. Our final
model, ChapGTP, is a masked language model that was trained for 200 epochs,
aided by a novel data augmentation technique called Automatic Task Formation.
We discuss in detail the performance of this model on the three evaluation
suites: BLiMP, (Super)GLUE, and MSGS. Furthermore, we present a wide range of
methods that were ultimately not included in the model, but may serve as
inspiration for training LMs in low-resource settings.