Mean BERTs make erratic language teachers: the effectiveness of latent
  bootstrapping in low-resource settings
David Samuel
This paper explores the use of latent bootstrapping, an alternative
self-supervision technique, for pretraining language models. Unlike the typical
practice of using self-supervision on discrete subwords, latent bootstrapping
leverages contextualized embeddings for a richer supervision signal. We conduct
experiments to assess how effective this approach is for acquiring linguistic
knowledge from limited resources. Specifically, our experiments are based on
the BabyLM shared task, which includes pretraining on two small curated corpora
and an evaluation on four linguistic benchmarks.