Convolutions Through the Lens of Tensor Networks
Felix Dangel
Despite their simple intuition, convolutions are more tedious to analyze than
dense layers, which complicates the generalization of theoretical and
algorithmic ideas. We provide a new perspective onto convolutions through
tensor networks (TNs) which allow reasoning about the underlying tensor
multiplications by drawing diagrams, and manipulating them to perform function
transformations, sub-tensor access, and fusion. We demonstrate this expressive
power by deriving the diagrams of various autodiff operations and popular
approximations of second-order information with full hyper-parameter support,
batching, channel groups, and generalization to arbitrary convolution
dimensions. Further, we provide convolution-specific transformations based on
the connectivity pattern which allow to re-wire and simplify diagrams before
evaluation. Finally, we probe computational performance, relying on established
machinery for efficient TN contraction. Our TN implementation speeds up a
recently-proposed KFAC variant up to 4.5x and enables new hardware-efficient
tensor dropout for approximate backpropagation.