Tune As You Scale: Hyperparameter Optimization For Compute Efficient
  Training
Abraham J. Fetterman Ellie Kitanidis Joshua Albrecht Zachary Polizzi Bryden Fogelman Maksis Knutins Bartosz Wróblewski James B. Simon Kanjun Qiu
Hyperparameter tuning of deep learning models can lead to order-of-magnitude
performance gains for the same amount of compute. Despite this, systematic
tuning is uncommon, particularly for large models, which are expensive to
evaluate and tend to have many hyperparameters, necessitating difficult
judgment calls about tradeoffs, budgets, and search bounds. To address these
issues and propose a practical method for robustly tuning large models, we
present Cost-Aware Pareto Region Bayesian Search (CARBS), a Bayesian
optimization algorithm that performs local search around the performance-cost
Pareto frontier. CARBS does well even in unbounded search spaces with many
hyperparameters, learns scaling relationships so that it can tune models even
as they are scaled up, and automates much of the "black magic" of tuning. Among
our results, we effectively solve the entire ProcGen benchmark just by tuning a
simple baseline (PPO, as provided in the original ProcGen paper). We also
reproduce the model size vs. training tokens scaling result from the Chinchilla
project (Hoffmann et al. 2022), while simultaneously discovering scaling laws
for every other hyperparameter, via an easy automated process that uses
significantly less compute and is applicable to any deep learning problem (not
just language models).