We prove a multiclass boosting theory for the ResNet architectures which simultaneously creates a new technique for multiclass boosting and provides a new algorithm for ResNet-style architectures.   Our proposed training algorithm, BoostResNet, is particularly suitable in non-differentiable architectures.   Our method only requires the relatively inexpensive sequential training of T "shallow ResNets". We prove that the training error decays exponentially with the depth T if the weak module classifiers that we train perform slightly better than some weak baseline.   In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition.   A generalization error bound based on margin theory is proved and suggests that ResNet could be resistant to overfitting using a network with l_1 norm bounded weights. Why do residual neural networks (ResNets) BID14 and the related highway networks BID35 work? And if we study closely why they work, can we come up with new understandings of how to train them and how to define working algorithms?Deep neural networks have elicited breakthrough successes in machine learning, especially in image classification and object recognition BID19 BID32 BID34 Zeiler & Fergus, 2014) in recent years. As the number of layers increases, the nonlinear network becomes more powerful, deriving richer features from input data. Empirical studies suggest that challenging tasks in image classification BID15 BID34 and object recognition BID7 BID8 BID12 BID23 often require "deep" networks, consisting of tens or hundreds of layers. Theoretical analyses have further justified the power of deep networks BID24 compared to shallow networks.However, deep neural networks are difficult to train despite their intrinsic representational power. Stochastic gradient descent with back-propagation (BP) BID20 and its variants are commonly used to solve the non-convex optimization problems. A major challenge that exists for training both shallow and deep networks is vanishing or exploding gradients BID1 BID9 . Recent works have proposed normalization techniques BID9 BID22 BID15 BID31 to effectively ease the problem and achieve convergence. In training deep networks, however, a surprising training performance degradation is observed BID35 BID14 : the training performance degrades rapidly with increased network depth after some saturation point. This training performance degradation is representationally surprising as one can easily construct a deep network identical to a shallow network by forcing any part of the deep network to be the same as the shallow network with the remaining layers functioning as identity maps. He et al. BID14 presented a residual network (ResNet) learning framework to ease the training of networks that are substantially deeper than those used previously. And they explicitly reformulate the layers as learning residual functions with reference to the layer inputs by adding identity loops to the layers. It is shown in BID10 that identity loops ease the problem of spurious local optima in shallow networks. BID35 introduce a novel architecture that enables the optimization of networks with virtually arbitrary depth through the use of a learned gating mechanism for regulating information flow.Empirical evidence overwhelmingly shows that these deep residual networks are easier to optimize than non-residual ones. Can we develop a theoretical justification for this observation? And does that justification point us towards new algorithms with better characteristics? Our proposed BoostResNet algorithm achieves exponentially decaying (with the depth T ) training error under the weak learning condition. BoostResNet is much more computationally efficient compared to end-to-end back-propagation in deep ResNet. More importantly, the memory required by BoostResNet is trivial compared to end-to-end back-propagation. It is particularly beneficial given the limited GPU memory and large network depth. Our learning framework is natural for nondifferentiable data. For instance, our learning framework is amenable to take weak learning oracles using tensor decomposition techniques. Tensor decomposition, a spectral learning framework with theoretical guarantees, is applied to learning one layer MLP in BID16 . We plan to extend our learning framework to non-differentiable data using general weak learning oracles. In neural network optimization, there are many commonly-used loss functions and criteria, e.g., mean squared error, negative log likelihood, margin criterion, etc. There are extensive works BID7 BID30 BID37 on selecting or modifying loss functions to prevent empirical difficulties such as exploding/vanishing gradients or slow learning BID0 . However, there are no rigorous principles for selecting a loss function in general. Other works consider variations of the multilayer perceptron (MLP) or convolutional neural network (CNN) by adding identity skip connections BID14 , allowing information to bypass particular layers. However, no theoretical guarantees on the training error are provided despite breakthrough empirical successes. Hardt et al. BID10 have shown the advantage of identity loops in linear neural networks with theoretical justifications; however the linear setting is unrealistic in practice.
We prove a multiclass boosting theory for the ResNet architectures which simultaneously creates a new technique for multiclass boosting and provides a new algorithm for ResNet-style architectures. Presents a boosting-style algorithm for training deep residual networks, a convergence analysis for training error, and a analysis of generalization ability. A learning method for ResNet using the boosting framework that decomposes the learning of complex networks and uses less computational costs. Authors propose the deep ResNet as a boosting algorithm, and they claim this is more efficient than standard end-to-end backropagation.