Neural networks have recently had a lot of success for many tasks. However, neural
 network architectures that perform well are still typically designed manually
 by experts in a cumbersome trial-and-error process. We propose a new method
 to automatically search for well-performing CNN architectures based on a simple
 hill climbing procedure whose operators apply network morphisms, followed
 by short optimization runs by cosine annealing. Surprisingly, this simple method
 yields competitive results, despite only requiring resources in the same order of
 magnitude as training a single network. E.g., on CIFAR-10, our method designs
 and trains networks with an error rate below 6% in only 12 hours on a single GPU;
 training for one day reduces this error further, to almost 5%. Neural networks have rapidly gained popularity over the last few years due to their success in a variety of tasks, such as image recognition BID16 , speech recognition and machine translation BID1 . In most cases, these neural networks are still designed by hand, which is an exhausting, time-consuming process. Additionally, the vast amount of possible configurations requires expert knowledge to restrict the search. Therefore, a natural goal is to design optimization algorithms that automate this neural architecture search.However, most classic optimization algorithms do not apply to this problem, since the architecture search space is discrete (e.g., number of layers, layer types) and conditional (e.g., the number of parameters defining a layer depends on the layer type). Thus, methods that rely on, e.g., differentiability or independent parameters are not applicable. This led to a growing interest in using evolutionary algorithms BID22 BID26 and reinforcement learning BID2 BID6 for automatically designing CNN architectures. Unfortunately, most proposed methods are either very costly (requiring hundreds or thousands of GPU days) or yield non-competitive performance.In this work, we aim to dramatically reduce these computational costs while still achieving competitive performance. Specifically, our contributions are as follows:• We propose a baseline method that randomly constructs networks and trains them with SGDR BID20 . We demonstrate that this simple baseline achieves 6%-7% test error on CIFAR-10, which already rivals several existing methods for neural archictecture search. Due to its simplicity, we hope that this baseline provides a valuable starting point for the development of more sophisticated methods in the future.• We formalize and extend the work on network morphisms BID27 BID6 in order to provide popular network building blocks, such as skip connections and batch normalization.• We propose Neural Architecture Search by Hillclimbing (NASH), a simple iterative approach that, at each step, applies a set of alternative network morphisms to the current network, trains the resulting child networks with short optimization runs of cosine annealing BID20 , and moves to the most promising child network. NASH finds and trains competitive architectures at a computational cost of the same order of magnitude as training a single network; e.g., on CIFAR-10, NASH finds and trains CNNs with an error rate below 6 % in roughly 12 hours on a single GPU. After one day the error is reduced to almost 5%. Models from different stages of our algorithm can be combined to achieve an error of 4.7 % within two days on a single GPU. On CIFAR-100 , we achieve an error below 24% in one day and get close to 20% after two days.• Our method is easy to use and easy to extend, so it hopefully can serve as a basis for future work.We first discuss related work in Section 2. Then, we formalize the concept of network morphisms in Section 3 and propose our architecture search methods based on them in Section 4. We evaluate our methods in Section 5 and conclude in Section 6. We proposed NASH, a simple and fast method for automated architecture search based on a hill climbing strategy, network morphisms, and training via SGDR. Experiments on CIFAR-10 and CIFAR-100 showed that our method yields competitive results while requiring considerably less computational resources than most alternative approaches. Our algorithm is easily extendable, e.g., by other network morphisms, evolutionary approaches for generating new models, other methods for cheap performance evaluation (such as, e.g., learning curve prediction BID15 or hypernetworks BID10 BID5 ), or better resource handling strategies (such as Hyperband BID18 ). In this sense, we hope that our approach can serve as a basis for the development of more sophisticated methods that yield further improvements of performance. B SOME MODELS
We propose a simple and efficent method for architecture search for convolutional neural networks. Proposes a neural architecture search method that achieves close to state-of-the-art accuracy on CIFAR10 and takes much less computational resources. Presents a method to search neural network architectures at the same time of training which dramatically saves training time and architecture searching time. Proposes variant of neural architecture search using network morphisms to define a search space using CNN architectures completing the CIFAR image classification task