Automatic Combination of Sample Selection Strategies for Few-Shot
  Learning
Branislav Pecher Ivan Srba Maria Bielikova Joaquin Vanschoren
In few-shot learning, such as meta-learning, few-shot fine-tuning or
in-context learning, the limited number of samples used to train a model have a
significant impact on the overall success. Although a large number of sample
selection strategies exist, their impact on the performance of few-shot
learning is not extensively known, as most of them have been so far evaluated
in typical supervised settings only. In this paper, we thoroughly investigate
the impact of 20 sample selection strategies on the performance of 5 few-shot
learning approaches over 8 image and 6 text datasets. In addition, we propose a
new method for automatic combination of sample selection strategies (ACSESS)
that leverages the strengths and complementary information of the individual
strategies. The experimental results show that our method consistently
outperforms the individual selection strategies, as well as the recently
proposed method for selecting support examples for in-context learning. We also
show a strong modality, dataset and approach dependence for the majority of
strategies as well as their dependence on the number of shots - demonstrating
that the sample selection strategies play a significant role for lower number
of shots, but regresses to random selection at higher number of shots.