conv_einsum: A Framework for Representation and Fast Evaluation of
  Multilinear Operations in Convolutional Tensorial Neural Networks
Tahseen Rabbani Jiahao Su Xiaoyu Liu David Chan Geoffrey Sangston Furong Huang
Modern ConvNets continue to achieve state-of-the-art results over a vast
array of vision and image classification tasks, but at the cost of increasing
parameters. One strategy for compactifying a network without sacrificing much
expressive power is to reshape it into a tensorial neural network (TNN), which
is a higher-order tensorization of its layers, followed by a factorization,
such as a CP-decomposition, which strips a weight down to its critical basis
components. Passes through TNNs can be represented as sequences of multilinear
operations (MLOs), where the evaluation path can greatly affect the number of
floating point operations (FLOPs) incurred. While functions such as the popular
einsum can evaluate simple MLOs such as contractions, existing implementations
cannot process multi-way convolutions, resulting in scant assessments of how
optimal evaluation paths through tensorized convolutional layers can improve
training speed. In this paper, we develop a unifying framework for representing
tensorial convolution layers as einsum-like strings and a meta-algorithm
conv_einsum which is able to evaluate these strings in a FLOPs-minimizing
manner. Comprehensive experiments, using our open-source implementation, over a
wide range of models, tensor decompositions, and diverse tasks, demonstrate
that conv_einsum significantly increases both computational and
memory-efficiency of convolutional TNNs.