A widely observed phenomenon in deep learning is the degradation problem: increasing
 the depth of a network leads to a decrease in performance on both test and training data. Novel architectures such as ResNets and Highway networks have addressed this issue by introducing various flavors of skip-connections or gating mechanisms. However, the degradation problem persists in the context of plain feed-forward networks. In this work we propose a simple method to address this issue. The proposed method poses the learning of weights in deep networks as a constrained optimization problem where the presence of skip-connections is penalized by Lagrange multipliers. This allows for skip-connections to be introduced during the early stages of training and subsequently phased out in a principled manner. We demonstrate the benefits of such an approach with experiments on MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100 where the proposed method is shown to greatly decrease the degradation effect (compared to plain networks) and is often competitive with ResNets. The representation view of deep learning suggests that neural networks learn an increasingly abstract representation of input data in a hierarchical fashion BID26 BID6 BID7 . Such representations may then be exploited to perform various tasks such as image classification, machine translation and speech recognition.A natural conclusion of the representation view is that deeper networks will learn more detailed and abstract representations as a result of their increased capacity. However, in the case of feed-forward networks it has been observed that performance deteriorates beyond a certain depth, even when the network is applied to training data. Recently, Residual Networks (ResNets; BID9 and Highway Networks BID22 have demonstrated that introducing various flavors of skip-connections or gating mechanisms makes it possible to train increasingly deep networks. However, the aforementioned degradation problem persists in the case of plain deep networks (i.e., networks without skip-connections of some form).A widely held hypothesis explaining the success of ResNets is that the introduction of skipconnections serves to improve the conditioning of the optimization manifold as well as the statistical properties of gradients employed during training. BID19 and BID21 show that the introduction of specially designed skip-connections serves to diagonalize the Fisher information matrix, thereby bringing standard gradient steps closer to the natural gradient. More recently, BID0 demonstrated that the introduction of skip-connections helps retain the correlation structure across gradients. This is contrary to the gradients of deep feed-forward networks, which resemble white noise. More generally, the skip-connections are thought to reduce the effects of vanishing gradients by introducing a linear term BID10 .The goal of this work is to address the degradation issue in plain feed-forward networks by leveraging some of the desirable optimization properties of ResNets. We approach the task of learning parameters for a deep network under the framework of constrained optimization. This strategy allows us to introduce skip-connections penalized by Lagrange multipliers into the architecture of our network. In our setting , skip-connections play an important role during the initial training of the network and are subsequently removed in a principled manner. Throughout a series of experiments we demonstrate that such an approach leads to improvements in generalization error when compared to architectures without skip-connections and is competitive with ResNets in some cases.The contributions of this work are as follows:• We propose alternative training strategy for plain feed-forward networks which reduces the degradation in performance as the depth of the network increases. The proposed method introduces skip-connections which are penalized by Lagrange multipliers. This allows for the presence of skip-connections to be iteratively phased out during training in a principled manner. The proposed method is thereby able to enjoy the optimization benefits associated with skip-connections during the early stages of training.• A number of benchmark datasets are used to demonstrate the empirical capabilities of the proposed method. In particular , the proposed method greatly reduces the degradation effect compared to plain networks and is on several occasions competitive with ResNets. This manuscript presents a simple method for training deep feed-forward networks which greatly reduces the degradation problem. In the past, the degradation issue has been successfully addressed via the introduction of skip-connections. As such, the goal of this work is to propose a new training regime which retains the optimization benefits associated with ResNets while ultimately phasing out skip-connections. This is achieved by posing network training as a constrained optimization problem where skip-connections are introduced during the early stages of training and subsequently phased out in a principled manner using Lagrange multipliers.Throughout a series of experiments we demonstrate that the performance of VAN networks is stable, displaying a far smaller drop in performance as depth increases and thereby largely mitigating the degradation problem.
Phasing out skip-connections in a principled manner avoids degradation in deep feed-forward networks. The authors present a new training strategy, VAN, for training very deep feed-forward networks without skip connections The paper introduces an architecture that linearly interpolates between ResNets and vanilla deep nets without skip connections.