We study the error landscape of deep linear and nonlinear neural networks with the squared error loss. Minimizing the loss of a deep linear neural network is a nonconvex problem, and despite recent progress, our understanding of this loss surface is still incomplete. For deep linear networks, we present necessary and sufficient conditions for a critical point of the risk function to be a global minimum. Surprisingly, our conditions provide an efficiently checkable test for global optimality, while such tests are typically intractable in nonconvex optimization. We further extend these results to deep nonlinear neural networks and prove similar sufficient conditions for global optimality, albeit in a more limited function space setting. Since the advent of AlexNet BID10 , deep neural networks have surged in popularity, and have redefined the state-of-the-art across many application areas of machine learning and artificial intelligence, such as computer vision, speech recognition, and natural language processing. However, a concrete theoretical understanding of why deep neural networks work well in practice remains elusive. From the perspective of optimization, a significant barrier is imposed by the nonconvexity of training neural networks. Moreover, it was proved by BID2 that training even a 3-node neural network to global optimality is NP-Hard in the worst case, so there is little hope that neural networks have properties that make global optimization tractable.Despite the difficulties of optimizing weights in neural networks, the empirical successes suggest that the local minima of their loss surfaces could be close to global minima; and several papers have recently appeared in the literature attempting to provide a theoretical justification for the success of these models. For example, by relating neural networks to spherical spin-glass models from statistical physics, BID3 provided some empirical evidence that the increase of size of neural networks makes local minima close to global minima.Another line of results BID16 BID14 BID15 BID13 provides conditions under which a critical point of the empirical risk is a global minimum. Such results roughly involve proving that if full rank conditions of certain matrices (as well as some additional technical conditions) are satisfied, derivative of the risk being zero implies loss being zero. However, these results are obtained under restrictive assumptions; for example, BID13 require the width of one of the hidden layers to be as large as the number of training examples. BID14 and BID15 require the product of widths of two adjacent layers to be at least as large as the number of training examples, meaning that the number of parameters in the model must grow rapidly as we have more training data available. Another recent paper BID4 provides a sufficient condition for global optimality when the neural network is composed of subnetworks with identical architectures connected in parallel and a regularizer is designed to control the number of parallel architectures.Towards obtaining a more precise characterization of the loss-surfaces, a valuable conceptual simplification of deep nonlinear networks is deep linear neural networks, in which all activation functions are linear and the output of the entire network is a chained product of weight matrices with the input vector. Although at first sight a deep linear model may appear overly simplistic, even its opti-mization is nonconvex, and only recently theoretical results on this problem have started emerging. Interestingly, already in 1989, BID0 showed that some shallow linear neural networks have no local minima. More recently, BID8 extended this result to deep linear networks and proved that any local minimum is also global while any other critical point is a saddle point. Subsequently, BID11 provided a simpler proof that any local minimum is also global, with fewer assumptions than BID8 . Motivated by the success of deep residual networks BID6 b) , BID5 investigated loss surfaces of deep linear residual networks and showed every critical point is a global minimum in a near-identity region; subsequently, Bartlett et al. (2017) extended this result to a nonlinear function space setting.
We provide efficiently checkable necessary and sufficient conditions for global optimality in deep linear neural networks, with some initial extensions to nonlinear settings. The paper gives conditions for the global optimality of the loss function of deep linear neural networks The paper gives theoretical results regarding the existence of local minima in the objective function of deep neural networks. Studies some theoretical properties of deep linear networks.