SigOpt Mulch: An Intelligent System for AutoML of Gradient Boosted Trees
Aleksei Sorokin Xinran Zhu Eric Hans Lee Bolong Cheng
Gradient boosted trees (GBTs) are ubiquitous models used by researchers,
machine learning (ML) practitioners, and data scientists because of their
robust performance, interpretable behavior, and ease-of-use. One critical
challenge in training GBTs is the tuning of their hyperparameters. In practice,
selecting these hyperparameters is often done manually. Recently, the ML
community has advocated for tuning hyperparameters through black-box
optimization and developed state-of-the-art systems to do so. However, applying
such systems to tune GBTs suffers from two drawbacks. First, these systems are
not \textit{model-aware}, rather they are designed to apply to a
\textit{generic} model; this leaves significant optimization performance on the
table. Second, using these systems requires \textit{domain knowledge} such as
the choice of hyperparameter search space, which is an antithesis to the
automatic experimentation that black-box optimization aims to provide. In this
paper, we present SigOpt Mulch, a model-aware hyperparameter tuning system
specifically designed for automated tuning of GBTs that provides two
improvements over existing systems. First, Mulch leverages powerful techniques
in metalearning and multifidelity optimization to perform model-aware
hyperparameter optimization. Second, it automates the process of learning
performant hyperparameters by making intelligent decisions about the
optimization search space, thus reducing the need for user domain knowledge.
These innovations allow Mulch to identify good GBT hyperparameters far more
efficiently -- and in a more seamless and user-friendly way -- than existing
black-box hyperparameter tuning systems.