NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification
  Tasks
Jean-Michel Attendu Jean-Philippe Corbeil
Finetuning large language models inflates the costs of NLU applications and
remains the bottleneck of development cycles. Recent works in computer vision
use data pruning to reduce training time. Pruned data selection with static
methods is based on a score calculated for each training example prior to
finetuning, which involves important computational overhead. Moreover, the
score may not necessarily be representative of sample importance throughout the
entire training duration. We propose to address these issues with a refined
version of dynamic data pruning, a curriculum which periodically scores and
discards unimportant examples during finetuning. Our method leverages an EL2N
metric that we extend to the joint intent and slot classification task, and an
initial finetuning phase on the full train set. Our results on the GLUE
benchmark and four joint NLU datasets show a better time-accuracy trade-off
compared to static methods. Our method preserves full accuracy while training
on 50% of the data points and reduces computational times by up to 41%. If we
tolerate instead a minor drop of accuracy of 1%, we can prune 80% of the
training examples for a reduction in finetuning time reaching 66%.