Sensitivity Analysis of RF+clust for Leave-one-problem-out Performance
  Prediction
Ana Nikolikj Michal Pluháček Carola Doerr Peter Korošec Tome Eftimov
Leave-one-problem-out (LOPO) performance prediction requires machine learning
(ML) models to extrapolate algorithms' performance from a set of training
problems to a previously unseen problem. LOPO is a very challenging task even
for state-of-the-art approaches. Models that work well in the easier
leave-one-instance-out scenario often fail to generalize well to the LOPO
setting. To address the LOPO problem, recent work suggested enriching standard
random forest (RF) performance regression models with a weighted average of
algorithms' performance on training problems that are considered similar to a
test problem. More precisely, in this RF+clust approach, the weights are chosen
proportionally to the distances of the problems in some feature space. Here in
this work, we extend the RF+clust approach by adjusting the distance-based
weights with the importance of the features for performance regression. That
is, instead of considering cosine distance in the feature space, we consider a
weighted distance measure, with weights depending on the relevance of the
feature for the regression model. Our empirical evaluation of the modified
RF+clust approach on the CEC 2014 benchmark suite confirms its advantages over
the naive distance measure. However, we also observe room for improvement, in
particular with respect to more expressive feature portfolios.