RADIN: Souping on a Budget
Thibaut Menes Olivier Risser-Maroix
Model Soups, extending Stochastic Weights Averaging (SWA), combine models
fine-tuned with different hyperparameters. Yet, their adoption is hindered by
computational challenges due to subset selection issues. In this paper, we
propose to speed up model soups by approximating soups performance using
averaged ensemble logits performances. Theoretical insights validate the
congruence between ensemble logits and weight averaging soups across any mixing
ratios. Our Resource ADjusted soups craftINg (RADIN) procedure stands out by
allowing flexible evaluation budgets, enabling users to adjust his budget of
exploration adapted to his resources while increasing performance at lower
budget compared to previous greedy approach (up to 4% on ImageNet).