A Unified Gaussian Process for Branching and Nested Hyperparameter
  Optimization
Jiazhao Zhang Ying Hung Chung-Ching Lin Zicheng Liu
Choosing appropriate hyperparameters plays a crucial role in the success of
neural networks as hyper-parameters directly control the behavior and
performance of the training algorithms. To obtain efficient tuning, Bayesian
optimization methods based on Gaussian process (GP) models are widely used.
Despite numerous applications of Bayesian optimization in deep learning, the
existing methodologies are developed based on a convenient but restrictive
assumption that the tuning parameters are independent of each other. However,
tuning parameters with conditional dependence are common in practice. In this
paper, we focus on two types of them: branching and nested parameters. Nested
parameters refer to those tuning parameters that exist only within a particular
setting of another tuning parameter, and a parameter within which other
parameters are nested is called a branching parameter. To capture the
conditional dependence between branching and nested parameters, a unified
Bayesian optimization framework is proposed. The sufficient conditions are
rigorously derived to guarantee the validity of the kernel function, and the
asymptotic convergence of the proposed optimization framework is proven under
the continuum-armed-bandit setting. Based on the new GP model, which accounts
for the dependent structure among input variables through a new kernel
function, higher prediction accuracy and better optimization efficiency are
observed in a series of synthetic simulations and real data applications of
neural networks. Sensitivity analysis is also performed to provide insights
into how changes in hyperparameter values affect prediction accuracy.