Large GPT-like Models are Bad Babies: A Closer Look at the Relationship
  between Linguistic Competence and Psycholinguistic Measures
Julius Steuer Marius Mosbach Dietrich Klakow
Research on the cognitive plausibility of language models (LMs) has so far
mostly concentrated on modelling psycholinguistic response variables such as
reading times, gaze durations and N400/P600 EEG signals, while mostly leaving
out the dimension of what Mahowald et al. (2023) described as formal and
functional linguistic competence, and developmental plausibility. We address
this gap by training a series of GPT-like language models of different sizes on
the strict version of the BabyLM pretraining corpus, evaluating on the
challenge tasks (BLiMP, GLUE, MSGS) and an additional reading time prediction
task. We find a positive correlation between LM size and performance on all
three challenge tasks, with different preferences for model width and depth in
each of the tasks. In contrast, a negative correlation was found between LM
size and reading time fit of linear mixed-effects models using LM surprisal as
a predictor, with the second-smallest LM achieving the largest log-likelihood
reduction over a baseline model without surprisal. This suggests that modelling
processing effort and linguistic competence may require an approach different
from training GPT-like LMs on a developmentally plausible corpus.