The goal of this paper is to demonstrate a method for tensorizing neural networks based upon an efficient way of approximating scale invariant quantum states, the Multi-scale Entanglement Renormalization Ansatz (MERA). We employ MERA as a replacement for linear layers in a neural network and test this implementation on the CIFAR-10 dataset. The proposed method outperforms factorization using tensor trains, providing greater compression for the same level of accuracy and greater accuracy for the same level of compression. We demonstrate MERA-layers with 3900 times fewer parameters and a reduction in accuracy of less than 1% compared to the equivalent fully connected layers.
 The curse of dimensionality is a major bottleneck in machine learning, stemming from the exponential growth of variables with the number of modes in a data set BID0 ). Typically state-of-the-art convolutional neural networks have millions or billions of parameters. However, previous work has demonstrated that representations stored in the network parameters can be highly compressed without significant reduction in network performance BID15 , BID3 , BID5 ). Determining the best network architecture for a given task remains an open problem.Descriptions of quantum mechanical systems raise a similar challenge; representing n d-dimensional particles requires a rank-n tensor whose memory cost scales as d n . Indeed, it was the promise of harnessing this that led Richard Feynman BID2 ) to suggest the possibility of quantum computation. In the absence of a quantum computer, however, one must use compressed representations of quantum states.A level of compression can be achieved by factorizing the tensorial description of the quantum wavefunction. Many such factorizations are possible, the optimal structure of the factorization being determined by the structure of correlations in the quantum system being studied. A revolution in quantum mechanics was made by realizing that the best way to characterize the distribution of correlations and information in a state is by a quantity known as entanglement -loosely the mutual quantum information between partitions of a quantum system BID1 ).This has led to many successful applications of tensorial approaches to problems in solid state physics and quantum chemistry over the past 25 years BID16 , BID7 ). Intriguing ideas have also emerged over the past few years attempting to bridge the successes of neural networks in machine learning with those of tensorial methods in quantum physics, both at a fundamental level BID10 , BID13 ), and as a practical tool for network design BID9 ). Recent work has suggested that entanglement itself is a useful quantifier of the performance of neural networks BID9 , BID11 The simplest factorization employed in quantum systems is known as the matrix product state BID16 ). In essence, it expresses the locality of information in certain quantum states. It has already been adopted to replace expensive linear layers in neural networks -in which context it has been independently termed tensor trains BID17 ). This led to substantial compression of neural networks with only a small reduction in the accuracy BID15 , BID3 ).Here we use a different tensor factorization -known as the Multi-scale Entanglement Renormalization Ansatz (MERA) -that encodes information in a hierarchical manner BID24 ). MERA works through a process of coarse graining or renormalization. There have been a number of papers looking at the relationship between renormalization and deep learning. MERA is a concrete realization of such a renormalization procedure BID25 ) and so possesses a multi-scale structure that one might anticipate in complex data. A number of works have utilized tree tensor network models that possess a similar hierarchical structure. However, they do not include the disentangler tensors that are essential if each layer of the MERA is to capture correlations on different length scales BID11 ).In this work we employ MERA as a replacement for linear layers in a neural network used to classify the CIFAR-10 dataset. Our results show that this performs better than the tensor train decomposition of the same linear layer, and gives better accuracy for the same level of compression and better compression for the same level of accuracy. In Section 2 we introduce factorizations of fully connected linear layers, starting with the tensor train factorization followed by a tree-like factorization and finally the MERA factorization. In Section 3 we discuss how this is employed as a replacement for a fully connected linear layer in deep learning networks. Section 4 gives our main results and we note connections with the existing literature in Section 5. Finally, in Section 6 we discuss some potential developments of the work. In this report we have replaced the linear layers of the standard neural network with tensorial MERA layers. The first step in achieving this involves expressing a linear layer as a tensor. This can be accomplished by taking a matrix W and reshaping it to be a higher dimensional array. For example, suppose W is d n by d n dimensional with components W AB . It can be transformed into a rank 2n tensor by mapping A to n elements A → i 1 , i 2 , ..., i n and B to another n elements B → j 1 , j 2 , ..., j n . In this case each of the elements of the new tensor will be of size d. FIG0 gives a graphical representation of this rank 2n tensor W i1,i2,... ,in j1,j2,...,jn . It is important to note that in this representation, the lines represent the indices of the tensors rather than weights. FIG0 We have shown that replacing the fully connected layers of a deep neural network with layers based upon the multi-scale entanglement renormalization ansatz can generate significant efficiency gains with only small reduction in accuracy. When applied to the CIFAR-10 data we found the fully connected layers can be replaced with MERA layers with 3900 times less parameters with a reduction in the accuracy of less than 1%. The model significantly outperformed compact fully connected layers with 70 − 100 times as many parameters. Moreover, it outperformed a similar replacement of the fully connected layers with tensor trains, both in terms of accuracy for a given compression and compression for a given accuracy.An added advantage -not explored here -is that a factorized layer can potentially handle much larger input data sets, thus enabling entirely new types of computation. Correlations across these large inputs can be handled much more efficiently by MERA than by tensor trains. Moreover, a compressed network may provide a convenient way to avoid over-fitting of large data sets. The compression achieved by networks with these factorized layers comes at a cost. They can take longer to train than networks containing the large fully connected layers due to the number of tensor contractions required to apply the factorized layer.Our results suggest several immediate directions for future inquiry. Firstly, there are some questions about how to improve the existing model. For example, before the MERA layer is used the input is reshaped into a rank-12 tensor. There isn't a well defined method for how to perform this reshaping optimally and some experimentation is necessary. The best way to initialize the MERA layers is also still an open question.The results presented here are a promising first step for using MERA in a more fundamental way. Since MERA can be viewed as a coarse graining procedure (as explained in Section 2), and image data is often well represented in a hierarchical manner, one possibility would be to simply train a two-dimensional MERA directly on an image dataset, with no reference to a neural network. In BID22 a similar idea was explored with matrix product states being trained directly on MNIST. An alternative possibility would be the replacement of just the convolutional layers of the network with a two-dimensional MERA. Both of these approaches would be closer in spirit to the fundamental ideas about the relationships between quantum physics and machine learning proposed in BID10 and BID13 .Additionally , there has been some work using entanglement measures to explore how correlations are distributed in deep neural networks, and then utilizing these in order to optimize the design of networks BID11 , BID9 ). It would be intriguing to explore such ideas using MERA, for example by using the concrete MERA model explored in this paper, or one of the more ambitious possibilities mentioned above.We end by noting two facts: any variational approximation to a quantum wavefunction can be used to construct a replacement for linear layers of a network. There are many examples and each may have its sphere of useful application. Moreover, quantum computers of the type being developed currently by several groups are precisely described by a type of tensor network (a finite-depth circuit -and one that may very soon be too large to manipulate classically) and could be used as direct replacement for linear layers in a hybrid quantum/classical neural computation scheme.
We replace the fully connected layers of a neural network with the multi-scale entanglement renormalization ansatz, a type of quantum operation which describes long range correlations.  In the paper the authors suggest to use MERA tensorization technique for compressing neural networks. A new parameterization of linear maps for neural network use, using a hierarchical factorization of the linear map that reduces the number of parameters while still allowing for relatively complex interactions to be modelled. Studies compressing feed forward layers using low rank tensor decompositions and explore a tree like decomposition