Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampling distributions.
 In particular, we propose the use of $k$-determinantal point processes in  hyperparameter optimization via random search. Compared to conventional uniform random search where hyperparameter settings are sampled independently, a $k$-DPP promotes diversity.   We describe an approach that transforms hyperparameter search spaces for efficient use with a $k$-DPP. In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from $k$-DPPs defined over spaces with a mixture of discrete and continuous dimensions. Our experiments show significant benefits over uniform random search  in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel. Hyperparameter values-regularization strength, model family choices like depth of a neural network or which nonlinear functions to use, procedural elements like dropout rates, stochastic gradient descent step sizes, and data preprocessing choices-can make the difference between a successful application of machine learning and a wasted effort. To search among many hyperparameter values requires repeated execution of often-expensive learning algorithms, creating a major obstacle for practitioners and researchers alike.In general, on request/iteration k, a hyperparameter searcher suggests a hyperparameter configuration x k , a worker trains a model using x k , and returns a validation loss of y k computed on a hold out set. In this work we say a hyperparameter searcher is open loop if x k depends only on {x i } k−1 i=1 ; examples include choosing x k uniformly at random BID4 , or x k coming from a low-discrepancy sequence (c.f., BID12 ). We say a searcher is closed loop if x k depends on both the past configurations and validation losses {(x i , y i )} k−1 i=1 ; examples include Bayesian optimization BID19 and recent reinforcement learning methods BID25 . Note that open loop methods can draw an infinite sequence of configurations before training a single model, whereas closed loop methods rely on validation loss feedback in order to make suggestions.While sophisticated closed loop selection methods have been shown to empirically identify good hyperparameter configurations faster (i.e., with fewer iterations) than open loop methods like random search, two trends have rekindled interest in embarrassingly parallel open loop methods: 1) modern deep learning models can take days or weeks to train with no signs of efficiency breakthroughs, and 2) the rise of cloud resources available to anyone that charge not by the number of machines, but by the number of CPU-hours used so that 10 machines for 100 hours costs the same as 1000 machines for 1 hour. This paper explores the landscape of open loop methods, identifying tradeoffs that are rarely considered, if at all acknowledged. While random search is arguably the most popular open loop method and chooses each x k independently of {x i } k−1 i=1 , it is by no means the only choice. In many ways uniform random search is the least interesting of the methods we will discuss because we will advocate for methods where x k depends on {x i } k−1 i=1 to promote diversity. In particular, we will focus on drawing {x i } k i=1 from a k-determinantal point process (DPP) BID16 . DPPs support real, integer, and categorical dimensions-any of which may have a tree structure-and have computationally efficient methods of drawing samples.Experimentally, we explore the use of our diversity-promoting open-loop hyperparameter optimization method based on k-DPP random search. We find that it significantly outperforms uniform random search in cases where the hyperparameter values have a large effect on performance.Open source implementations of both our hyperparameter optimization algorithm (as an extension to the hyperopt package BID5 ) and the MCMC algorithm introduced in Algorithm 2 will be released upon publication. We have explored open loop hyperparameter optimization built on sampling from k-DPPs. We described how to construct k-DPPs over hyperparameter search spaces, and showed that sampling from these retains the attractive parallelization capabilities of random search. Our experiments demonstrate that, under a limited computation budget, on a number of realistic hyperparameter optimization problems, these approaches perform better than sampling uniformly at random. As we increase the difficulty of our hyperparameter optimization problem (i.e., as values which lead to good model Average best-found model accuracy by iteration when training a convolutional neural network on the "Stable" search space (defined in Section 5.2), averaged across 50 trials of hyperparameter optimization, with k = 20. Discretizing the space reduces the accuracy found for both uniform sampling and k-DPP-RBF, but in both cases k-DPP-RBF finds better optima than uniform sampling.evaluations become more scarce) the improvement over sampling uniformly at random increases. An open-source implementation of our method is available.
Driven by the need for parallelizable, open-loop hyperparameter optimization methods, we propose the use of $k$-determinantal point processes in  hyperparameter optimization via random search. Proposes using the k-DPP to select candidate points in hyperparameter searches. The authors propose k-DPP as an open loop method for hyperparameter optimization and provide its empirical study and comparison with other methods. Considers non-sequential and uninformed hyperparameter search using determinantal point processes, which are probability distributions over subsets of a ground set with the property that subsets with more 'diverse' elements haev higher probability