Multi-Objective Optimization of Performance and Interpretability of
  Tabular Supervised Machine Learning Models
Lennart Schneider Bernd Bischl Janek Thomas
We present a model-agnostic framework for jointly optimizing the predictive
performance and interpretability of supervised machine learning models for
tabular data. Interpretability is quantified via three measures: feature
sparsity, interaction sparsity of features, and sparsity of non-monotone
feature effects. By treating hyperparameter optimization of a machine learning
algorithm as a multi-objective optimization problem, our framework allows for
generating diverse models that trade off high performance and ease of
interpretability in a single optimization run. Efficient optimization is
achieved via augmentation of the search space of the learning algorithm by
incorporating feature selection, interaction and monotonicity constraints into
the hyperparameter search space. We demonstrate that the optimization problem
effectively translates to finding the Pareto optimal set of groups of selected
features that are allowed to interact in a model, along with finding their
optimal monotonicity constraints and optimal hyperparameters of the learning
algorithm itself. We then introduce a novel evolutionary algorithm that can
operate efficiently on this augmented search space. In benchmark experiments,
we show that our framework is capable of finding diverse models that are highly
competitive or outperform state-of-the-art XGBoost or Explainable Boosting
Machine models, both with respect to performance and interpretability.