Assessing the Generalizability of a Performance Predictive Model
Ana Nikolikj Gjorgjina Cenikj Gordana Ispirova Diederick Vermetten Ryan Dieter Lang Andries Petrus Engelbrecht Carola Doerr Peter Korošec Tome Eftimov
A key component of automated algorithm selection and configuration, which in
most cases are performed using supervised machine learning (ML) methods is a
good-performing predictive model. The predictive model uses the feature
representation of a set of problem instances as input data and predicts the
algorithm performance achieved on them. Common machine learning models struggle
to make predictions for instances with feature representations not covered by
the training data, resulting in poor generalization to unseen problems. In this
study, we propose a workflow to estimate the generalizability of a predictive
model for algorithm performance, trained on one benchmark suite to another. The
workflow has been tested by training predictive models across benchmark suites
and the results show that generalizability patterns in the landscape feature
space are reflected in the performance space.