We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-affine spline operators (MASOs). We prove that RNNs using piecewise affine and convex nonlinearities can be written as a simple piecewise affine spline operator. The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper. First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time. Second, we show that the affine slope parameter of an RNN corresponds to an input-specific template, from which we can interpret an RNN as performing a simple template matching (matched filtering) given the input. Third, by carefully examining the MASO RNN affine mapping, we prove that using a random initial hidden state corresponds to an explicit L2 regularization of the affine parameters, which can mollify exploding gradients and improve generalization. Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions. In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets. Recurrent neural networks (RNNs) are a powerful class of models for processing sequential inputs and a basic building block for more advanced models that have found success in challenging problems involving sequential data, including sequence classification (e.g., sentiment analysis BID30 , sequence generation (e.g., machine translation BID1 ), speech recognition BID10 , and image captioning BID22 . Despite their success, however, our understanding of how RNNs work remains limited. For instance, an attractive theoretical result is the universal approximation property that states that an RNN can approximate an arbitrary function BID28 BID29 BID11 . These classical theoretical results have been obtained primarily from the dynamical system BID29 BID28 and measure theory BID11 perspectives. These theories provide approximation error bounds but unfortunately limited guidance on applying RNNs and understanding their performance and behavior in practice.In this paper, we provide a new angle for understanding RNNs using max-affine spline operators (MASOs) BID21 BID12 ) from approximation theory. The piecewise affine approximations made by compositions of MASOs provide a new and useful framework to study neural networks. For example, BID4 ; BID2 have provided a detailed analysis in the context of feedforward networks. Here, we go one step further and find new insights and interpretations from the MASO perspective for RNNs. We will see that the input space partitioning and matched filtering links developed in BID4 ; BID2 extend to RNNs and yield interesting insights into their inner workings. Moreover, the MASO formulation of RNNs enables us to theoretically justify the use of a random initial hidden state to improve RNN performance.For concreteness, we focus our analysis on a specific class of simple RNNs BID8 with piecewise affine and convex nonlinearities such as the ReLU BID9 . RNNs with such nonlinearities have recently gained considerable attention due to their ability to combat the exploding gradient problem; with proper initialization BID19 BID33 and clever parametrization of the recurrent weight BID0 BID39 BID16 BID15 BID24 BID13 , these RNNs achieve performance on par with more complex ones such as LSTMs. Below is a summary of our key contributions. Contribution 1. We prove that an RNN with piecewise affine and convex nonlinearities can be rewritten as a composition of MASOs, making it a piecewise affine spline operator with an elegant analytical form (Section 3).Contribution 2. We leverage the partitioning of piecewise affine spline operators to analyze the input space partitioning that an RNN implicitly performs. We show that an RNN calculates a new, high-dimensional representation (the partition code) of the input sequence that captures informative underlying characteristics of the input. We also provide a new perspective on RNN dynamics by visualizing the evolution of the RNN input space partitioning through time (Section 4).Contribution 3. We show the piecewise affine mapping in an RNN associated with a given input sequence corresponds to an input-dependent template, from which we can interpret the RNN as performing greedy template matching (matched filtering) at every RNN cell (Section 5).Contribution 4. We rigorously prove that using a random (rather than zero) initial hidden state in an RNN corresponds to an explicit regularizer that can mollify exploding gradients. We show empirically that such a regularization improves RNN performance (to state-of-the-art) on four datasets of different modalities (Section 6). We have developed and explored a novel perspective of RNNs in terms of max-affine spline operators (MASOs). RNNs with piecewise affine and convex nonlinearities are piecewise affine spline operators with a simple, elegant analytical form. The connections to input space partitioning (vector quantization) and matched filtering followed immediately. The spline viewpoint also suggested that the typical zero initial hidden state be replaced with a random one that mollifies the exploding gradient problem and improves generalization performance.There remain abundant promising research directions. First, we can extend the MASO RNN framework following BID3 to cover more general networks like gated RNNs (e.g, GRUs, LSTMs) that employ the sigmoid nonlinearity, which is neither piecewise affine nor convex. Second, we can apply recent random matrix theory results BID23 to the affine parameter A RNN (e.g., the change of the distribution of its singular values during training) to understand RNN training dynamics. t th time step of a discrete time-serie, DISPLAYFORM0 x Concatenation of the whole length T time-serie: DISPLAYFORM1 Output/prediction associated with input x y n True label (target variable) associated with the nth time-serie example x n . For classification y n ∈ {1, . . . , C}, C > 1; For regression y n ∈ R C , C ≥ 1 DISPLAYFORM2 Output of an RNN cell at layer and time step t; Alternatively, input to an RNN cell at layer + 1 and time step t − 1 DISPLAYFORM3 Concatenation of hidden state h ( ,t) of all time steps at layer : DISPLAYFORM4 Concatenated input to an RNN cell at layer and time step t: DISPLAYFORM5 th layer RNN weight associated with the input h ( ,t−1) from the previous time step: DISPLAYFORM6 th layer RNN weight associated with the input h ( −1,t) from the previous layer: DISPLAYFORM7 Bias of the last fully connected layer: DISPLAYFORM8 Pointwise nonlinearity in an RNN (assumed to be piecewise affine and convex in this paper) σ Standard deviation of noise injected into the initial hidden state h DISPLAYFORM9 MASO formula of the RNN activation σ(·) at layer and time step t: DISPLAYFORM10 MASO parameters of an RNN at layer and time step t: DISPLAYFORM11
We provide new insights and interpretations of RNNs from a max-affine spline operators perspective. Rewrites equations of Elman RNN in terms of so-called max-affine spline operators Provide a novel approach toward understanding RNNs using max-affline spline operators (MASO) by rewriting them with piecewise affine and convex activations MASOs The authors build upon max-affine spline operator interpetation of a substantial class of deep networks, focusing on Recurrent Neural Networks using noise in initial hidden state acts as regularization