Convolutional Neural Networks (CNNs) filter the input data using a series of spatial convolution operators with compactly supported stencils and point-wise nonlinearities.
 Commonly, the convolution operators couple features from all channels.
 For wide networks, this leads to immense computational cost in the training of and prediction with CNNs.
 In this paper, we present novel ways to parameterize the convolution more efficiently, aiming to decrease the number of parameters in CNNs and their computational complexity.
 We propose new architectures that use a sparser coupling between the channels and thereby reduce both the number of trainable weights and the computational cost of the CNN.
 Our architectures arise as new types of residual neural network (ResNet) that can be seen as discretizations of a Partial Differential Equations (PDEs) and thus have predictable theoretical properties. Our first architecture involves a convolution operator with a special sparsity structure, and is applicable to a large class of CNNs. Next, we present an architecture that can be seen as a discretization of a diffusion reaction PDE, and use it with three different convolution operators. We outline in our experiments that the proposed architectures,  although considerably reducing the number of trainable weights, yield comparable accuracy to existing CNNs that are fully coupled in the channel dimension.
 Convolutional Neural Networks (CNNs) BID18 are among the most effective machine learning approaches for processing structured high-dimensional input data and are indispensable in, e.g., in recognition tasks involving speech BID22 and image data BID16 . The essential idea behind CNNs is to replace some or all of the affine linear transformations in a neural network by convolution operators that are typically parameterized using small-dimensional stencils. This has a number of benefits including the increase of computational efficiency of the network due to the sparse connections between features, and a considerable reduction in the number of weights since stencils are shared across the whole feature map BID6 .In practical applications of CNNs, the features can be grouped into channels whose number is associated with the width of the network. This gives one several opportunities to define interactions between the different channels. Perhaps , the most common approach in CNNs is to fully couple features across channels BID7 BID6 BID16 . Following this approach, the number of convolution operators at a layer is proportional to the product of the number of input and output channels. Given that performing convolutions is often the computationally most expensive part in training of and prediction with CNNs and the number of channels is large in many applications, this scaling can be problematic for wide architectures or high-dimensional data. Another disadvantage of this type of architecture is the number of weights. Indeed, for deep neural networks, the number of weights that are associated with a wide network can easily reach millions and beyond. This makes the deployment of such networks challenging, especially on devices with limited memory.In this paper, we propose four novel ways to parameterize CNNs more efficiently, based on ideas from Partial Differential Equations (PDEs). Our goal is to dramatically reduce the number of weights in the networks and the computational costs of training and evaluating the CNNs. One ides, similar to BID13 , is to use spatial convolutions for each channel individually and add Table 1 : Cost comparison of different convolution layers for an image with n pixels, stencil of size m × m, and c input and output channels. RD denotes a reaction-diffusion architecture.no. of weights computational costs fully-coupled DISPLAYFORM0 1 × 1 convolutions to impose coupling between them. Our architectures are motivated by the interpretation of residual neural networks (ResNets) BID11 BID18 as time-dependent nonlinear PDEs BID24 . More specifically, we consider a simple Reaction-Diffusion (RD) model, that can model highly nonlinear processes. We derive new architectures by discretizing this continuous model, using 1×1 convolutions as a reaction term, together with cheap forms of a spatial convolution, that are similar to a depth-wise convolution in the number of parameters and cost. This spatial convolution acts similarly to a linear diffusion term that smooths the feature channels. Since the networks we propose originate in continuous models they have distinct theoretical properties that can be predicted using the standard theory of ODEs and PDEs BID0 .Our first approach is designed to be employed in any existing CNN layer with equal number of input and output channels. We simply replace the traditional fully coupled convolution with a linear sum of depth-wise and 1 × 1 convolution, like a mask that can be placed on a traditional convolution in any existing CNN. Our second "explicit" RD architecture applies the operators separately with a non-linear activation function operating only following the 1 × 1 convolution, as the non-linear reaction part of the diffusion reaction equation. The third architecture is more unique . To improve the stability of the forward propagation and increase the spatial coupling of the features, we propose a semi-implicit scheme for the forward propagation through the network. Unlike traditional CNN operators, the semi-implicit scheme applies an inverse of the depth-wise (block diagonal) convolution preceded by a non-linear step involving the 1 × 1 convolution. This way, the scheme couples all the pixels in the image in one layer, even though we are using a depth-wise 3 × 3 convolution. The inverse of the convolution operator can be efficiently computed using Fast Fourier Transforms (FFT) and over the channels and kernels.The last idea is to replace the depth-wise convolution structure with a circulant connectivity between the channels. This is motivated by the interpretation of the features as tensors and follows the definition of an efficient tensor product in BID14 whose associated tensor singular value decomposition has been successfully used for image classification in BID20 . The circulant structure renders the number of trainable convolution stencils proportional to the width of the layer. Using periodic boundary conditions in the other feature dimensions, this convolution can be computed efficiently by extending the FFT-based convolutions in BID19 BID26 along the channel dimension, which reduces the cost from O(c 2 ) to O(c log c) where c is the number of channels. Table 1 compares the number of weights and the computational complexity associated with the forward propagation through a layer for the standard and reduced architectures. In the table we assume that the explicit RD architecture is directly computed without using FFT, but the FFT-based implementation, which is necessary for the implicit scheme, can also be used for the explicit one.Our architectures pursue a similar goal than the recently proposed MobileNet architectures that are also based on a mix of 1 × 1 and "depth-wise" convolutions BID13 BID25 . The MobileNet architecture involves with significantly less parameters, and in particular avoids the fully coupled convolutions, except for 1 × 1 convolutions which are cheaper in both computational cost and number of parameters. What sets our work apart from these architectures is that our architectures can be seen as discretization of PDEs, which allows to control their stability and offers new ways for their analysis.The remainder of the paper is organized as follows. We first describe the mathematical formulation of the supervised classification problem with deep residual neural networks used in this paper. Subsequently, we propose the novel parameterizations of CNNs, describe their efficient implementation, and their computational costs. We perform experiments using the CIFAR10, CIFAR 100, and STL10 datasets and demonstrate that the performance of the new architectures, despite a considerable reduction in the number of trainable weights, is comparable to residual neural networks using fully-coupled convolutions. Finally, we summarize and conclude the paper. We present four new convolution models with the common goal of reducing the number of parameters and computational costs of CNNs. To this end, we propose alternative ways to the traditional full coupling of channels, and thereby obtain architectures that involve fewer expensive convolutions, avoid redundancies in the network parametrization, and thereby can be deployed more widely. Our work is similar to that of BID13 BID25 . However, our unique angle is the close relation of our architectures to continuous models given in terms of PDEs that are well understood. This highlights stability of our CNNs and paves the way toward more extensive theory.Our numerical experiments for image classification show that the new architectures can be almost as effective as more expensive fully coupled CNN architectures. We expect that our architectures will be able to replace the traditional convolutions in classification of audio and video, and also in other tasks that are treated with CNNs. It is important to realize that our new architectures become even more advantageous for 3D or 4D problems, e.g., when analyzing time series of medical or geophysical images. In these cases, the cost of each convolution is much more expensive and the computational complexity makes using 3D CNNs difficult. Here, also the number of weights imposes challenges when using computational hardware with moderate memory.
This paper introduces efficient and economic parametrizations of convolutional neural networks motivated by partial differential equations  Introduces four "low cost" alternatives to the standard convolution operation that can be used in place of the standard convolution operation to reduce their computational complexity. This paper introduces methods for reducing the computational cost of CNN implementations, and introduces new parameterizations of CNN like architectures that limit parameter coupling. The paper proposes a PDE-based perspective to understand and parameterize CNNs