Convolution neural networks typically consist of many convolutional layers followed by several fully-connected layers.   While convolutional layers map between high-order activation tensors, the fully-connected layers operate on flattened activation vectors.   Despite its success, this approach has notable drawbacks. Flattening discards the multi-dimensional structure of the activations, and the fully-connected layers require a large number of parameters. 
 We present two new techniques to address these problems.   First, we introduce tensor contraction layers which can replace the ordinary fully-connected layers in a neural network. Second, we introduce tensor regression layers, which express the output of a neural network as a low-rank multi-linear mapping from a high-order activation tensor to the softmax layer.   Both the contraction and regression weights are learned end-to-end by backpropagation. By imposing low rank on both, we use significantly fewer parameters.   Experiments on the ImageNet dataset show that applied to the popular VGG and ResNet architectures, our methods significantly reduce the number of parameters in the fully connected layers (about 65% space savings) while negligibly impacting accuracy. Many natural datasets exhibit pronounced multi-modal structure. We represent audio spectrograms as 2nd-order tensors (matrices) with modes corresponding to frequency and time. We represent images as third-order tensors with modes corresponding to width, height and the color channels. Videos are expressed as 4th-order tensors, and the signal processed by an array of video sensors can be described as a 5th-order tensor. A broad array of multi-modal data can be naturally encoded as tensors. Tensor methods extend linear algebra to higher order tensors and are promising tools for manipulating and analyzing such data.The mathematical properties of tensors have long been the subject of theoretical study. Previously, in machine learning, data points were typically assumed to be vectors and datasets to be matrices. Hence, spectral methods, such as matrix decompositions, have been popular in machine learning. Recently, tensor methods, which generalize these techniques to higher-order tensors, have gained prominence. One class of broadly useful techniques within tensor methods are tensor decompositions, which have been studied for learning latent variables BID0 .Deep Neural Networks (DNNs) frequently manipulate high-order tensors: in a standard deep convolutional Neural Network (CNN) for image recognition, the inputs and the activations of convolutional layers are 3 rd -order tensors. And yet, to wit, most architectures output predictions by first flattening the activations tensors and then connecting to the output neurons via one or more fullyconnected layers. This approach presents several issues: we lose multi-modal information during the flattening process and the fully-connected layers require a large number of parameters.In this paper, we propose Tensor Contraction Layers (TCLs) and Tensor Regression Layers (TRLs) as end-to-end trainable components of neural networks. In doing so, we exploit multilinear structure without giving up the power and flexibility offered by modern deep learning methods. By replacing fully-connected layers with tensor contractions, we can aggregate long-range spatial information while preserving multi-modal structure. Moreover, by enforcing low rank, we can significantly reduce the number of parameters needed with minimal impact on accuracy.Our proposed TRL represent the regression weights through the factors of a low-rank tensor decomposition. The TRL obviates the need for flattening when generating output. By combining tensor regression with tensor contraction, we further increase efficiency. Augmenting the VGG and ResNet architectures, we demonstrate improved performance on the ImageNet dataset despite significantly reducing the number of parameters (almost by 65%). This is the first paper that presents an end-to-end trainable architecture that retains the multi-dimensional tensor structure throughout the network.Related work: Several recent papers apply tensor decomposition to deep learning. BID16 propose using CP decomposition to speed up convolutional layers. BID13 take a pre-trained network and apply tensor (Tucker) decomposition on the convolutional kernel tensors and then fine-tune the resulting network. BID23 propose weight sharing in multi-task learning and BID2 propose sharing residual units. These contributions are orthogonal to ours and can be applied together. BID17 use the Tensor-Train (TT) format to impose low-rank tensor structure on weights. However, they still retain the fully-connected layers for the output, while we present an end-to-end tensorized network architecture.Despite the success of DNNs, many open questions remain as to why they work so well and whether they really need so many parameters. Tensor methods have emerged as promising tools of analysis to address these questions and to better understand the success of deep neural networks. BID3 , for example, use tensor methods as tools of analysis to study the expressive power of CNNs. BID5 derive sufficient conditions for global optimality and optimization of non-convex factorization problems, including tensor factorization and deep neural network training. Other papers investigate tensor methods as tools for devising neural network learning algorithms with theoretical guarantees of convergence BID20 BID11 b) . Several prior papers address the power of tensor regression to preserve natural multi-modal structure and learn compact predictive models BID4 BID18 BID25 BID24 . However, these works typically rely on analytical solutions and require manipulating large tensors containing the data. They are usually used for small dataset or require to downsampled datasets or extract compact features prior to fitting the model, and do not scale to large datasets such as ImageNet.To our knowledge, no prior work combines tensor contraction or tensor regression with deep learning in an end-to-end trainable fashion. Unlike fully-connected layers, TCLs and TRLs obviate the need to flatten input tensors. Our experiments demonstrate that by imposing a low-rank constraint on the weights of the regression, we can learn a low-rank manifold on which both the data and the labels lie. The result is a compact network, that achieves similar accuracies with many fewer parameters. Going forward, we plan to apply the TCL and TRL to more network architectures. We also plan to leverage recent work BID21 on extending BLAS primitives to avoid transpositions needed when computing tensor contractions.
We propose tensor contraction and low-rank tensor regression layers to preserve and leverage the multi-linear structure throughout the network, resulting in huge space savings with little to no impact on performance. This paper proposes new layer architectures of neural networks using a low-rank representation of tensors This paper incorporates tensor decomposition and tensor regression into CNN by using a new tensor regression layer.