Too Much Information: Keeping Training Simple for BabyLMs
Lukas Edman Lisa Bylinina
This paper details the work of the University of Groningen for the BabyLM
Challenge. We follow the idea that, like babies, language models should be
introduced to simpler concepts first and build off of that knowledge to
understand more complex concepts. We examine this strategy of
simple-then-complex through a variety of lenses, namely context size,
vocabulary, and overall linguistic complexity of the data. We find that only
one, context size, is truly beneficial to training a language model. However
this simple change to context size gives us improvements of 2 points on average
on (Super)GLUE tasks, 1 point on MSGS tasks, and 12\% on average on BLiMP
tasks. Our context-limited model outperforms the baseline that was trained on
10$\times$ the amount of data.