Regularized boosting with an increasing coefficient magnitude stop
  criterion as meta-learner in hyperparameter optimization stacking ensemble
Laura Fdez-Díaz José Ramón Quevedo Elena Montañés
In Hyperparameter Optimization (HPO), only the hyperparameter configuration
with the best performance is chosen after performing several trials, then,
discarding the effort of training all the models with every hyperparameter
configuration trial and performing an ensemble of all them. This ensemble
consists of simply averaging the model predictions or weighting the models by a
certain probability. Recently, other more sophisticated ensemble strategies,
such as the Caruana method or the stacking strategy has been proposed. On the
one hand, the Caruana method performs well in HPO ensemble, since it is not
affected by the effects of multicollinearity, which is prevalent in HPO. It
just computes the average over a subset of predictions with replacement. But it
does not benefit from the generalization power of a learning process. On the
other hand, stacking methods include a learning procedure since a meta-learner
is required to perform the ensemble. Yet, one hardly finds advice about which
meta-learner is adequate. Besides, some meta-learners may suffer from the
effects of multicollinearity or need to be tuned to reduce them. This paper
explores meta-learners for stacking ensemble in HPO, free of hyperparameter
tuning, able to reduce the effects of multicollinearity and considering the
ensemble learning process generalization power. At this respect, the boosting
strategy seems promising as a stacking meta-learner. In fact, it completely
removes the effects of multicollinearity. This paper also proposes an implicit
regularization in the classical boosting method and a novel non-parametric stop
criterion suitable only for boosting and specifically designed for HPO. The
synergy between these two improvements over boosting exhibits competitive and
promising predictive power performance compared to other existing meta-learners
and ensemble approaches for HPO other than the stacking ensemble.