On the Effects of Randomness on Stability of Learning with Limited
  Labelled Data: A Systematic Literature Review
Branislav Pecher Ivan Srba Maria Bielikova
Learning with limited labelled data, such as few-shot learning, meta-learning
or transfer learning, aims to effectively train a model using only small amount
of labelled samples. However, these approaches were observed to be excessively
sensitive to the effects of uncontrolled randomness caused by non-determinism
in the training process. The randomness negatively affects the stability of the
models, leading to large variance in results across training runs. When such
instability is disregarded, it can unintentionally, but unfortunately also
intentionally, create an imaginary perception of research progress. Recently,
this area started to attract a research attention and the number of relevant
studies is continuously growing. In this survey, we provide a comprehensive
overview of 134 papers addressing the effects of randomness on the stability of
learning with limited labelled data. We distinguish between four main tasks
addressed in the papers (investigate/evaluate; determine; mitigate;
benchmark/compare/report randomness effects), providing findings for each one.
Furthermore, we identify and discuss seven challenges and open problems
together with possible directions to facilitate further research. The ultimate
goal of this survey is to emphasise the importance of this growing research
area, which so far has not received appropriate level of attention.