Recently Neural Architecture Search (NAS) has aroused great interest in both academia and industry, however it remains challenging because of its huge and non-continuous search space. Instead of applying evolutionary algorithm or reinforcement learning as previous works, this paper proposes a Direct Sparse Optimization NAS (DSO-NAS) method. In DSO-NAS, we provide a novel model pruning view to NAS problem. In specific, we start from a completely connected block, and then introduce scaling factors to scale the information flow between operations. Next, we impose sparse regularizations to prune useless connections in the architecture. Lastly, we derive an efficient and theoretically sound optimization method to solve it. Our method enjoys both advantages of differentiability and efficiency, therefore can be directly applied to large datasets like ImageNet. Particularly, On CIFAR-10 dataset, DSO-NAS achieves an average test error 2.84%, while on the ImageNet dataset DSO-NAS achieves 25.4% test error under 600M FLOPs with 8 GPUs in 18 hours. With no doubt, Deep Neural Networks (DNN) have been the engines for the AI renaissance in recent years. Dating back to 2012, DNN based methods have refreshed the records for many AI applications, such as image classification BID16 ; BID37 ; BID8 ), speech recognition ; BID4 ) and Go Game BID33 ). Considering its amazing representation power, DNNs have shifted the paradigm of these applications from manually designing the features and stagewise pipelines to end-to-end learning. Although DNNs have liberated researchers from such feature engineering, another tedious work has emerged -"network engineering". In most cases, the neural networks need to be designed based on the specific tasks, which again leads to endless hyperparameters tuning and trails. Therefore, designing a suitable neural network architecture still requires considerable amounts of expertise and experience.To democratize the techniques, Neural Architecture Search (NAS) or more broadly, AutoML has been proposed. There are mainly two streams for NAS: The first one is to follow the pioneering work BID45 , which proposed a reinforcement learning algorithm to train a Recurrent Neural Network (RNN) controller that generates coded architectures ; BID29 ). The second one is the evolutionary algorithm, which iteratively evaluates and proposes new models for evaluation BID30 ; BID36 ). Despite their impressive performance, the search processes are incredibly resource-hungry and unpractical for large datasets like ImageNet, though some acceleration methods have been proposed BID44 ; BID29 ). Very recently, DARTS BID22 ) proposed a gradient-based method in which the connections are selected by a softmax classifier. Although DARTS achieves impressive performance with great acceleration, its search space is still limited to fix-length coding and blocksharing search as in previous works.In this work, we take another view to tackle these problems. We reformulate NAS as pruning the useless connections from a large network which contains the complete network architecture hypothesis space. Thus only one single model is trained and evaluated. Since the network structure is directly optimized during training, we call our method Direct Sparse Optimization NAS (DSO-NAS). We further demonstrate that this sparse regularized problem can be efficiently optimized by a modified accelerated proximal gradient method opposed to the inefficient reinforcement learning or revolutionary search. Notably, DSO-NAS is much simpler than the existing search methods as it unifies the neural network weight learning and architecture search into one single optimization problem. DSO-NAS does not need any controller BID45 ; ; BID29 ) or performance predictor BID21 ) or relaxation of the search space BID45 ; ; BID29 ; BID22 ). As a result of the efficiency and simplicity, DSO-NAS first demonstrate that NAS can be directly applied to large datasets like ImageNet with no block structure sharing. Our experiments show that DSO-NAS can achieve 2.84% average test error on CIFAR-10, as well as top-1 error 25.4% on ImageNet with FLOPs (the number of multiply-adds) under 600M.In summary, our contributions can be summarized as follows:• We propose a novel model pruning formulation for neural architecture search based on sparse optimization. Only one model needs to be trained during the search.• We propose a theoretically sound optimization method to solve this challenging optimization problem both effectively and efficiently.• We demonstrate the results of our proposed method are competitive or better than other NAS methods, while significantly simplifying and accelerating the search process. Neural Architecture Search has been the core technology for realizing AutoML. In this paper, we have proposed a Direct Sparse Optimization method for NAS. Our method is appealing to both academic research and industrial practice in two aspects: First, our unified weight and structure learning method is fully differentiable in contrast to most previous works. It provides a novel model pruning view to the NAS problem. Second, the induced optimization method is both efficient and effective. We have demonstrated state-of-the-art performance on both CIFAR and ILSVRC2012 image classification datasets, with affordable cost (single machine in one day).In the future, we would like to incorporate hardware features for network co-design, since the actual running speed of the same network may highly vary across different hardware because of cache size, memory bandwidth, etc. We believe our proposed DSO-NAS opens a new direction to pursue such objective. It could push a further step to AutoML for everyone's use.
single shot neural architecture search via direct sparse optimization Presents an architecture search method where connections are removed with sparse regularization. This paper proposes Direct Sparse Optimization, which is a method to obtain neural architectures on specific problems, at a reasonable computational cost. This paper proposes a neural architecture search method based on a direct sparse optimization