On "Scientific Debt" in NLP: A Case for More Rigour in Language Model
  Pre-Training Research
Made Nindyatama Nityasya Haryo Akbarianto Wibowo Alham Fikri Aji Genta Indra Winata Radityo Eko Prasojo Phil Blunsom Adhiguna Kuncoro
This evidence-based position paper critiques current research practices
within the language model pre-training literature. Despite rapid recent
progress afforded by increasingly better pre-trained language models (PLMs),
current PLM research practices often conflate different possible sources of
model improvement, without conducting proper ablation studies and principled
comparisons between different models under comparable conditions. These
practices (i) leave us ill-equipped to understand which pre-training approaches
should be used under what circumstances; (ii) impede reproducibility and credit
assignment; and (iii) render it difficult to understand: "How exactly does each
factor contribute to the progress that we have today?" We provide a case in
point by revisiting the success of BERT over its baselines, ELMo and GPT-1, and
demonstrate how -- under comparable conditions where the baselines are tuned to
a similar extent -- these baselines (and even-simpler variants thereof) can, in
fact, achieve competitive or better performance than BERT. These findings
demonstrate how disentangling different factors of model improvements can lead
to valuable new insights. We conclude with recommendations for how to encourage
and incentivize this line of work, and accelerate progress towards a better and
more systematic understanding of what factors drive the progress of our
foundation models today.