What Happens When Small Is Made Smaller? Exploring the Impact of
  Compression on Small Data Pretrained Language Models
Busayo Awobade Mardiyyah Oduwole Steven Kolawole
Compression techniques have been crucial in advancing machine learning by
enabling efficient training and deployment of large-scale language models.
However, these techniques have received limited attention in the context of
low-resource language models, which are trained on even smaller amounts of data
and under computational constraints, a scenario known as the "low-resource
double-bind." This paper investigates the effectiveness of pruning, knowledge
distillation, and quantization on an exclusively low-resourced, small-data
language model, AfriBERTa. Through a battery of experiments, we assess the
effects of compression on performance across several metrics beyond accuracy.
Our study provides evidence that compression techniques significantly improve
the efficiency and effectiveness of small-data language models, confirming that
the prevailing beliefs regarding the effects of compression on large, heavily
parameterized models hold true for less-parameterized, small-data models.