Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use \emph{ad hoc} gating mechanisms.   Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues.
	
 We prove that learnable gates in a recurrent model formally provide \emph{quasi-invariance to general time transformations} in the input data. We recover part of the LSTM architecture from a simple axiomatic approach.
	
 This result leads to a new way of initializing gate biases in LSTMs and GRUs. Experimentally, this new \emph{chrono initialization} is shown to greatly improve learning of long term dependencies, with minimal implementation effort.

 The self loop feedback gating mechanism of recurrent networks has been derived from first principles via a postulate of invariance to time warpings. Gated connections appear to regulate the local time constants in recurrent models. With this in mind, the chrono initialization, a principled way of initializing gate biases in LSTMs, has been introduced. Experimentally, chrono initialization is shown to bring notable benefits when facing long term dependencies.A ADDITIONAL EXPERIMENTS On the generalization capacity of recurrent architectures. We proceeded to test the generalization properties of RNNs, leaky RNNs and chrono RNNs on the pure warping experiments presented in Section 3. For each of the architectures, a recurrent network with 64 recurrent units is trained for 3 epochs on a variable warping task with warps between 1 and 50. Each network is then tested on warped sequences, with warps between 100 and an increasingly big maximum warping. Results are summarized in Figure 5 .All networks display reasonably good, but not perfect, generalization. Even with warps 10 times longer than the training set warps, the networks still have decent accuracy, decreasing from 100% to around 75%.Interestingly , plain RNNs and gated RNNs display a different pattern: overall, gated RNNs perform better but their generalization performance decreases faster with warps eight to ten times longer than those seen during training, while plain RNN never have perfect accuracy, below 80% even within the training set range, but have a flatter performance when going beyond the training set warp range.Pixel level classification: MNIST and pMNIST. This task, introduced in BID15 , consists in classifying images using a recurrent model. The model is fed pixels one by one, from top to bottom, left to right, and has to output a probability distribution for the class of the object in the image.We evaluate standard and chrono initialization on two image datasets: MNIST (LeCun et al., 1999) and permuted MNIST, that is, MNIST where all images have undergone the same pixel permutation.LSTMs with 512 hidden units are used. Once again, standard initialization sets forget biases to 1, and the chrono initialization parameter is set to the length of the input sequences, max = 784. Results on the validation set are provided in Figure 6 . On non-permuted MNIST, there is no clear difference , even though the best validation error is obtained with chrono initialization. On permuted MNIST, chrono initialization performs better, with a best validation result of 96.3%, while standard initialization obtains a best validation result of 95.4%.Next character prediction on text8. Chrono initialization is benchmarked against standard initialization on the character level text8 dataset BID17 . Text8 is a 100M character formatted text sample from Wikipedia. BID21 's train-valid-test split is used: the first 90M characters are used as training set, the next 5M as validation set and the last 5M as test set.The exact same setup as in BID3 ) is used, with the code directly taken from there. Namely: LSTMs with 2000 units, trained with Adam BID13 with learning rate 10 −3 , batches of size 128 made of non-overlapping sequences of length 180, and gradient clipping at 1.0. Weights are orthogonally initialized, and recurrent batch normalization BID3 ) is used.Chrono initialization with max = 8 is compared to standard = 1 initialization. Results are presented in FIG4 . On the validation set, chrono initialization uniformly outperforms standard initialization by a small margin. On the test set, the compression rate is 1.37 with chrono initialization, versus 1.38 for standard initialization.8 This same slight difference is observed on two independent runs.Our guess is that , on next character prediction, with moderately sized networks, short term dependencies dominate, making the difference between standard and chrono initialization relatively small.Next word prediction on Penn Treebank. To attest for the resilience of chrono initialization to more complex models than simple LSTMs, we train on word level Penn Treebank BID21 using the best deep RHN network from BID25 . All hyperparameters are taken from of BID25 . For the chrono bias initialization, a single bias vector is sampled according to ∼ log( (1, max )), the carry gate bias vectors of all layers are initialized to − , and the transform gate biases to . max is chosen to be 11 (because this gives an average bias initialization close to the value 2 from (Zilly et al., 2016)).9 . Without further hyperparameter search and with a single run, we obtain test results similar to BID25 , with a test perplexity of 6.54.
Proves that gating mechanisms provide invariance to time transformations. Introduces and tests a new initialization for LSTMs from this insight. Paper links recurrent network deisgn and its effect on how the network reacts to time transformations, and uses this to develop a simple bias initialization scheme.