Are Compressed Language Models Less Subgroup Robust?
Leonidas Gee Andrea Zugarini Novi Quadrianto
To reduce the inference cost of large language models, model compression is
increasingly used to create smaller scalable models. However, little is known
about their robustness to minority subgroups defined by the labels and
attributes of a dataset. In this paper, we investigate the effects of 18
different compression methods and settings on the subgroup robustness of BERT
language models. We show that worst-group performance does not depend on model
size alone, but also on the compression method used. Additionally, we find that
model compression does not always worsen the performance on minority subgroups.
Altogether, our analysis serves to further research into the subgroup
robustness of model compression.