Machine learning models are usually tuned by nesting optimization of model weights inside the optimization of hyperparameters.   We give a method to collapse this nested optimization into joint stochastic optimization of both weights and hyperparameters.   Our method trains a neural network to output approximately optimal weights as a function of hyperparameters.   We show that our method converges to locally optimal weights and hyperparameters for sufficiently large hypernets.   We compare this method to standard hyperparameter optimization strategies and demonstrate its effectiveness for tuning thousands of hyperparameters. Hyperparameter λ Training and validation loss of a neural net, estimated by crossvalidation (crosses) or by a hypernet (lines), which outputs 7, 850-dimensional network weights. The training and validation loss can be cheaply evaluated at any hyperparameter value using a hypernet. Standard cross-validation requires training from scratch each time.Model selection and hyperparameter tuning is a major bottleneck in designing predictive models. Hyperparameter optimization can be seen as a nested optimization: The inner optimization finds model parameters w which minimizes the training loss LTrain given hyperparameters λ. The outer optimization chooses λ to minimize a validation loss LValid. : DISPLAYFORM0 Standard practice in machine learning solves (1) by gradient-free optimization of hyperparameters, such as grid search, random search, or Bayesian optimization. Each set of hyperparameters is evaluated by reinitializing weights and training the model to completion. This is wasteful, since it trains the model from scratch each time, even if the hyperparameters change a small amount. Hyperband BID14 and freezethaw Bayesian optimization (Swersky et al., 2014 ) resume model training and do not waste this effort. Furthermore, gradient-free optimization scales poorly beyond 10 or 20 dimensions.How can we avoid re-training from scratch each time?We usually estimate the parameters with stochastic optimization, but the true optimal parameters are a deterministic function of the hyperparameters λ: DISPLAYFORM1 We propose to learn this function. Specifically , we train a neural network with inputs of hyperparameters, and outputs of an approximately optimal set of weights given the hyperparameters.This provides two major benefits: First, we can train the hypernet to convergence using stochastic gradient descent, denoted SGD, without training any particular model to completion. Second, differentiating through the hypernet allows us to optimize hyperparameters with gradient-based stochastic optimization.P a ra m e te r w H y p e r p a r a m e t e r λ Loss L Train (w, In this paper, we:• Presented algorithms that efficiently learn a differentiable approximation to a best-response without nested optimization.• Showed empirically that hypernets can provide a better inductive bias for hyperparameter optimization than Gaussian processes fit directly to the validation loss.• Gave a theoretical justification that sufficiently large networks will learn the best-response for all hyperparameters it is trained against.We hope that this initial exploration of stochastic hyperparameter optimization will inspire further refinements, such as hyper-regularization methods, or uncertainty-aware exploration using Bayesian hypernetworks. A EXTRA EXPERIMENTS A.1 OPTIMIZING 10 HYPERPARAMETERS Here, we optimize a model with 10 hyperparameters, in which a separate L 2 weight decay is applied to the weights for each digit class in a linear regression model to see if we can optimize medium-sized models. The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiments. A linear hypernet is used, resulting in 86, 350 hyper-weights. Algorithm 3 is compared against random search and .Figure 8, right, shows that our method converges more quickly and to a better optimum than either alternative method, demonstrating that medium-sized hyperparameter optimization problems can be solved with Algorithm 3. Figure 8 : Validation and test losses during hyperparameter optimization. A separate L 2 weight decay is applied to the weights of each digit class, resulting in 10 hyperparameters. The weights w φ * are output by the hypernet for current hyperparameterλ, while random losses are for the best result of a random search. Hypernetwork-based optimization converges faster than random search or Bayesian optimization. We also observe significant overfitting of the hyperparameters on the validation set, which may be reduced be introducing hyperhyperparameters (parameters of the hyperparameter prior). The runtime includes the inner optimization for gradient-free approaches so that equal cumulative computational time is compared for each method.Factors affecting this include removing the overhead of constructing tuples of hyperparameters and optimized weights, viewing more hyperparameter samples, or having a better inductive bias from learning weights.
We train a neural network to output approximately optimal weights as a function of hyperparameters. 
Hyper-networks for hyper-parameter optimization in neural networks.