Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture. Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network. As a result, it is difficult to know a priori how successful a GRU-RNN will perform on a given data set. In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system.
 We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits. In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory. These findings were then experimentally verified in two dimensions by means of time series prediction. Recurrent neural networks (RNNs) have been widely used to capture and utilize sequential structure in natural and artificial languages, speech, video, and various other forms of time series. The recurrent information flow within RNN implies that the data seen in the past has influence on the current state of the RNN, forming a mechanism for having memory through (nonlinear) temporal traces. Unfortunately, training vanilla RNNs (which allow input data to directly interact with the hidden state) to capture long-range dependences within a sequence is challenging due to the vanishing gradient problem BID8 . Several special RNN architectures have been proposed to mitigate this issue, notably the long short-term memory (LSTM; BID9 ) which explicitly guards against unwanted corruption of the information stored in the hidden state until necessary. Recently, a simplification of the LSTM called the gated recurrent unit (GRU; BID1 ) has become wildly popular in the machine learning community thanks to its performance in machine translation BID0 , speech BID16 , music BID2 , video BID4 , and extracting nonlinear dynamics underlying neural data BID15 . As a variant of the vanilla LSTM, GRUs incorporate the use of forget gates, but lack an output gate BID5 . While this feature reduces the number of required parameters, LSTM has been shown to outperform GRU on neural machine translation BID0 . In addition, certain mechanistic tasks, specifically unbounded counting, come easy to LSTM networks but not to GRU networks BID18 . Despite these empirical findings, we lack systematic understanding of the internal time evolution of GRU's memory structure and its capability to represent nonlinear temporal dynamics.In general, a RNN can be written as h t+1 = f (h t , x t ) where x t is the current input in a sequence indexed by t, f is a point-wise nonlinear function, and h t represents the hidden memory state that carries all information responsible for future output. In the absence of input, the hidden state h t can evolve over time on its own: DISPLAYFORM0 where f (·) := f (·, 0) for notational simplicity. In other words, we can consider the temporal evolution of memory stored within RNN as a trajectory of a dynamical system defined by (1). Then we can use dynamical systems theory to investigate the fundamental limits in the expressive power of RNNs in terms of their temporal features. We develop a novel theoretical framework to study the dynamical features fundamentally attainable, in particular, given the particular form of GRU. We then validate the theory by training GRUs to predict time series with prescribed dynamics. Our analysis shows the rich but limited classes of dynamics the GRU can approximate in one, two, and arbitrary dimensions. We developed a new theoretical framework to analyze GRUs as a continuous dynamical system, and showed that two GRUs can exhibit a variety of expressive dynamic features, such as limit cycles, homoclinic orbits, and a substantial catalog of stability structures and bifurcations. However, we also showed that finitely many GRUs cannot exhibit the dynamics of an arbitrary continuous attractor. These claims were then experimentally verified in two dimensions. We believe these findings also unlock new avenues of research on the trainability of recurrent neural networks. Although we have analyzed GRUs only in 1-and 2-dimensions in near exhaustive, we believe that the insights extends to higher-dimensions. We leave rigorous analysis of higher-dimensional GRUs as future work.A CONTINUOUS TIME SYSTEM DERIVATION We begin with the fully gated GRU as a discrete time system, where the input vector x t has been set equal to zero, as depicted in FORMULA0 - FORMULA0 , where is the Hadamard product, and σ is the sigmoid function. DISPLAYFORM0 We recognize that (15) is a forward Euler discretization of a continuous time dynamical system. This allows us to consider the underlying continuous time dynamics on the basis of the discretization. The following steps are a walk through of the derivation:Since z t is a bounded function on R ∀t, there exists a functionz t , such that z t +z t = 1 at each time step (due to the symmetry of z t ,z t is the result of vertically flipping z t about 0.5, the midpoint of its range). As such, we can rewrite (15) withz t as depicted in FORMULA0 . DISPLAYFORM1 Let h(t) ≡ h t−1 . As a result, we can sayz t ≡z(t) and r t ≡ r(t), as depicted in FORMULA7 . DISPLAYFORM2 Dividing both sides of the equation by ∆t yields (24). DISPLAYFORM3 If we take the limit as ∆t → 0, we get the analogous continuous time system to (13) -(15), DISPLAYFORM4 whereḣ ≡
We classify the the dynamical features one and two GRU cells can and cannot capture in continuous time, and verify our findings experimentally with k-step time series prediction.  The authors analyse GRUs with hidden sizes of one and two as continuous-time dynamical systems, claiming that the expressive power of the hidden state representation can provide prior knowledge on how well a GRU will perform on a given dataset This paper analyzes GRUs from a dynamical systems perspective, and shows that 2d GRUs can be trained to adopt a variety of fixed points and can approximate line attractors, but cannot mimic a ring attractor. Converts GRU equations into continuous time and uses theory and experiemnts to study 1- and 2-dimensional GRU networks and showcase every variety of dynamical topology available in these systems