On Sensitivity of Learning with Limited Labelled Data to the Effects of
  Randomness: Impact of Interactions and Systematic Choices
Branislav Pecher Ivan Srba Maria Bielikova
While learning with limited labelled data can improve performance when the
labels are lacking, it is also sensitive to the effects of uncontrolled
randomness introduced by so-called randomness factors (e.g., varying order of
data). We propose a method to systematically investigate the effects of
randomness factors while taking the interactions between them into
consideration. To measure the true effects of an individual randomness factor,
our method mitigates the effects of other factors and observes how the
performance varies across multiple runs. Applying our method to multiple
randomness factors across in-context learning and fine-tuning approaches on 7
representative text classification tasks and meta-learning on 3 tasks, we show
that: 1) disregarding interactions between randomness factors in existing works
caused inconsistent findings due to incorrect attribution of the effects of
randomness factors, such as disproving the consistent sensitivity of in-context
learning to sample order even with random sample selection; and 2) besides
mutual interactions, the effects of randomness factors, especially sample
order, are also dependent on more systematic choices unexplored in existing
works, such as number of classes, samples per class or choice of prompt format.