Parameter Optimization with Conscious Allocation (POCA)
Joshua Inman Tanmay Khandait Giulia Pedrielli Lalitha Sankar
The performance of modern machine learning algorithms depends upon the
selection of a set of hyperparameters. Common examples of hyperparameters are
learning rate and the number of layers in a dense neural network. Auto-ML is a
branch of optimization that has produced important contributions in this area.
Within Auto-ML, hyperband-based approaches, which eliminate poorly-performing
configurations after evaluating them at low budgets, are among the most
effective. However, the performance of these algorithms strongly depends on how
effectively they allocate the computational budget to various hyperparameter
configurations. We present the new Parameter Optimization with Conscious
Allocation (POCA), a hyperband-based algorithm that adaptively allocates the
inputted budget to the hyperparameter configurations it generates following a
Bayesian sampling scheme. We compare POCA to its nearest competitor at
optimizing the hyperparameters of an artificial toy function and a deep neural
network and find that POCA finds strong configurations faster in both settings.