Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps. We introduce a simple stochastic algorithm (\textit{h}-detach) that is specific to LSTM optimization and targeted towards addressing this problem. Specifically, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed. Based on the hypothesis that these components carry information about long term dependencies (which we show empirically), their suppression can prevent LSTMs from capturing them. Our algorithm\footnote{Our code is available at https://github.com/bhargav104/h-detach. } prevents gradients flowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better. We show significant improvements over vanilla LSTM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization using our modification of LSTM gradient on various benchmark datasets. Recurrent Neural Networks (RNNs) BID25 ; BID4 ) are a class of neural network architectures used for modeling sequential data. Compared to feed-forward networks, the loss landscape of recurrent neural networks are much harder to optimize. Among others, this difficulty may be attributed to the exploding and vanishing gradient problem BID8 BID2 BID24 which is more severe for recurrent networks and arises due to the highly ill-conditioned nature of their loss surface. This problem becomes more evident in tasks where training data has dependencies that exist over long time scales.Due to the aforementioned optimization difficulty, variants of RNN architectures have been proposed that aim at addressing these problems. The most popular among such architectures that are used in a wide number of applications include long short term memory (LSTM, BID9 ) and gated recurrent unit (GRU, Chung et al. (2014) ) networks, which is a variant of LSTM with forget gates BID5 . These architectures mitigate such difficulties by introducing a linear temporal path that allows gradients to flow more freely across time steps. BID0 on the other hand try to address this problem by parameterizing a recurrent neural network to have unitary transition matrices based on the idea that unitary matrices have unit singular values which prevents gradients from exploding/vanishing.Among the aforementioned RNN architectures, LSTMs are arguably most widely used (for instance they have more representational power compared with GRUs BID31 ) and it remains a hard problem to optimize them on tasks that involve long term dependencies. Examples of such tasks are copying problem BID2 BID24 , and sequential MNIST (Le Figure 1 : The computational graph of a typical LSTM. Here we have omitted the inputs x i for convenience. The top horizontal path through the cell state units c t s is the linear temporal path which allows gradients to flow more freely over long durations. The dotted blue crosses along the computational paths denote the stochastic process of blocking the flow of gradients though the h t states (see Eq 2) during the back-propagation phase of LSTM. We call this approach h-detach. et al., 2015) , which are designed in such a way that the only way to produce the correct output is for the model to retain information over long time scales.The goal of this paper is to introduce a simple trick that is specific to LSTM optimization and improves its training on tasks that involve long term dependencies. To achieve this goal, we write out the full back-propagation gradient equation for LSTM parameters and split the composition of this gradient into its components resulting from different paths in the unrolled network. We then show that when LSTM weights are large in magnitude, the gradients through the linear temporal path (cell state) get suppressed (recall that this path was designed to allow smooth gradient flow over many time steps). We show empirical evidence that this path carries information about long term dependencies (see section 3.5) and hence gradients from this path getting suppressed is problematic for such tasks. To fix this problem, we introduce a simple stochastic algorithm that in expectation scales the individual gradient components, which prevents the gradients through the linear temporal path from being suppressed. In essence, the algorithm stochastically prevents gradient from flowing through the h-state of the LSTM (see figure 1) , hence we call it h-detach. Using this method, we show improvements in convergence/generalization over vanilla LSTM optimization on the copying task, transfer copying task, sequential and permuted MNIST, and image captioning. In section 3.5 we showed that LSTMs trained with h-detach are stable even without gradient clipping. We caution that while this is true, in general the gradient magnitude depends on the value of detaching probability used in h-detach. Hence for the general case, we do not recommend removing gradient clipping.When training stacked LSTMs, there are two ways in which h-detach can be used: 1) detaching the hidden state of all LSTMs simultaneously for a given time step t depending on the stochastic variable ξ t ) stochastically detaching the hidden state of each LSTM separately. We leave this for future work.h-detach stochastically blocks the gradient from flowing through the hidden states of LSTM. In corollary 1, we showed that in expectation, this is equivalent to dampening the gradient components from paths other than the cell state path. We especially chose this strategy because of its ease of implementation in current auto-differentiation libraries. Another approach to dampen these gradient components would be to directly multiply these components with a dampening factor. This feature is currently unavailable in these libraries but may be an interesting direction to look into. A downside of using this strategy though is that it will not reduce the amount of computation similar to h-detach (although it will not increase the amount of computation compared with vanilla LSTM either). Regularizing the recurrent weight matrices to have small norm can also potentially prevent the gradient components from the cell state path from being suppressed but it may also restrict the representational power of the model.Given the superficial similarity of h-detach with dropout, we outline the difference between the two methods. Dropout randomly masks the hidden units of a network during the forward pass (and can be seen as a variant of the stochastic delta rule BID6 ). Therefore, a common view of dropout is training an ensemble of networks BID30 . On the other hand, our method does not mask the hidden units during the forward pass. It instead randomly blocks the gradient component through the h-states of the LSTM only during the backward pass and does not change the output of the network during forward pass. More specifically, our theoretical analysis shows the precise behavior of our method: the effect of h-detach is that it changes the update direction used for descent which prevents the gradients through the cell state path from being suppressed.We would also like to point out that even though we show improvements on the image captioning task, it does not fit the profile of a task involving long term dependencies that we focus on. We believe the reason why our method leads to improvements on this task is that the gradient components from the cell state path are important for this task and our theoretical analysis shows that h-detach prevents these components from getting suppressed compared with the gradient components from the other paths. On the same note, we also tried our method on language modeling tasks but did not notice any benefit. We proposed a simple stochastic algorithm called h-detach aimed at improving LSTM performance on tasks that involve long term dependencies. We provided a theoretical understanding of the method using a novel analysis of the back-propagation equations of the LSTM architecture. We note that our method reduces the amount of computation needed during training compared to vanilla LSTM training. Finally, we empirically showed that h-detach is robust to initialization, makes the convergence of LSTM faster, and/or improves generalization compared to vanilla LSTM (and other existing methods) on various benchmark datasets. . The next T − 1 entries are set to a 8 , which constitutes a delay. The next single entry is a 9 , which represents a delimiter, which should indicate to the algorithm that it is now required to reproduce the initial 10 input tokens as output. The remaining 10 input entries are set to a 8 . The target sequence consists of T + 10 repeated entries of a 8 , followed by the first 10 entries of the input sequence in exactly the same order. DISPLAYFORM0 Here denotes the element-wise product, also called the Hadamard product. σ denotes the sigmoid activation function. DISPLAYFORM1 For any ∈ {f, g, o, i}, define E (w) to be a matrix of size dim(h t ) × dim([h t ; x t ]). We set all the elements of this matrix to 0s if if w is not an element of W . Further, if w = (W ) kl , then (E (w)) kl = 1 and (E (w)) k l = 0 for all (k , l ) = (k, l). DISPLAYFORM2 Lemma 1 Let us assume w is an entry of the matrix DISPLAYFORM3 Proof By chain rule of total differentiation, DISPLAYFORM4 We note that, DISPLAYFORM5 DISPLAYFORM6 Recall that h t = o t tanh(c t ), and thus DISPLAYFORM7 Using the previous Lemma as well as the above notation, we get DISPLAYFORM8 DISPLAYFORM9 Then, z t = (A t + B t )z t−1In other words, where all the symbols used to define A t and B t are defined in notation 1.Proof By Corollary 2, we get DISPLAYFORM10 Similarly by Corollary 3, we get DISPLAYFORM11 Thus we have DISPLAYFORM12 Applying this formula recursively proves the claim.Note: Since A t has 0 n 's in the second column of the block matrix representation, it ignores the contribution of z t coming from h t−1 , whereas B t (having non-zero block matrices only in the second column of the block matrix representation) only takes into account the contribution coming from h t−1 . Hence A t captures the contribution of the gradient coming from the cell state c t−1 . T andz t be the analogue of z t when applying h-detach with probability p during back-propagation. Then, z t = (A t + ξ t B t )(A t−1 + ξ t−1 B t−1 ) . . . (A 2 + ξ 2 B 2 )z 1 where ξ t , ξ t−1 , . . . , ξ 2 are i.i.d. Bernoulli random variables with probability p of being 1, A t and B t and are same as defined in theorem 1.Proof Replacing DISPLAYFORM13 Iterating this formula gives, z t = (A t + ξ t B t )(A t−1 + ξ t−1 B t−1 ) . . . (A 3 + ξ 3 B 3 )z 2Corollary 4 E[z t ] = (A t + pB t )(A t−1 + pB t−1 ) . . . (A 3 + pB 3 )z 2It suffices to take the expectation both sides, and use independence of ξ t 's.
A simple algorithm to improve optimization and handling of long term dependencies in LSTM The paper introduces a simple stochastic algorithm called h-detach that is specific to LSTM optimization and targeted towards addressing this problem. Proposes a simple modification to the training process of the LSTM to facilitate gradient propogation along cell states, or the "linear temporal path"