We propose the dense RNN, which has the fully connections from each hidden state to multiple preceding hidden states of all layers directly. As the density of the connection increases, the number of paths through which the gradient flows can be increased. It increases the magnitude of gradients, which help to prevent the vanishing gradient problem in time. Larger gradients, however, can also cause exploding gradient problem. To complement the trade-off between two problems, we propose an attention gate, which controls the amounts of gradient flows. We describe the relation between the attention gate and the gradient flows by approximation. The experiment on the language modeling using Penn Treebank corpus shows dense connections with the attention gate improve the model’s performance. In order to analyze sequential data, it is important to choose an appropriate model to represent the data. Recurrent neural network (RNN), as one of the model capturing sequential data, has been applied to many problems such as natural language , machine translation BID0 , speech recognition BID6 . There are two main research issues to improve the RNNs performance: 1) vanishing and exploding gradient problems and 2) regularization.The vanishing and exploding gradient problems occur as the sequential data has long-term dependency BID10 BID18 . One of the solutions is to add gate functions such as the long short-term memory (LSTM) and gated recurrent unit (GRU). The LSTM has additional gate functions and memory cells BID11 . The gate function can prevent the gradient from being vanished during back propagation through time. Gated recurrent unit (GRU) has similar performance with less gate functions BID1 .The part of sequential data whose boundary to distinguish the consecutive other parts, has the hierarchical structures. To handle the hierarchy, the model should capture the multiple timescales. In hierarchical multiple recurrent neural network (HM-RNN, Chung et al. (2016) ), the boundary information is also learned by implementing three operations such as update, copy and flush operator. In clockwork RNN BID14 , the hidden states are divided into multiple sub-modules, which act with different periods to capture multiple timescales. As all previous states within the recurrent depth do not always affect the next state, memory-augmented neural network (MANN, BID7 ) uses the memory to remember previous states and retrieve some of previous states if necessary.The basic way to handle multiple timescales along with preventing the vanishing gradient problem is to increases both of feedforward depth and recurrent depth to capture multiple timescales. Feedforward depth is the longest path from the input layer to the output layer. Recurrent depth is the longest path from arbitrary hidden state at time t to same hidden sate at time t + t . Increasing feedforward depth means stacking multiple recurrent layers deeply. It can capture fast and slow changing components in the sequential data BID19 BID3 BID9 . The low level layer in the stacked RNN captures short-term dependency. As the layer is higher , the aggregated information from lower layer is abstracted. Thus, as the layer is higher, the capacity to model long-term dependency increases. The number of nonlinearities in the stacked RNN, however, is proportional to the number of unfolded time steps regardless of the feedforward depth. Thus, the simple RNN and stacked RNN act identically in terms of long run.Increasing recurrent depth also increases the capability to capture long-term dependency in the data. The hidden state in vanilla RNN has only connection to previous time step's hidden state in the same layer. Adding the connections to multiple previous time steps hidden states can make the shortcut paths, which alleviates the vanishing problem. Nonlinear autoregressive exogenous model (NARX) handles the vanishing gradient problem by adding direct connections from the distant past in the same layer BID15 . Similarly, higher-order RNN (HO-RNN ) has the direct connections to multiple previous states with gating to each time step BID20 . Unlike other recurrent models that use one connection between two consecutive time steps, the recurrent highway network (RHN) adds multiple connections with sharing parameters between transitions in the same layer BID23 .The vanilla RNN has only one path connected with previous hidden states. Thus, it is hard to apply standard dropout technique for regularization as the information is being diluted during training of long-term sequences. By selecting the same dropout mask for feedforward , recurrent connections, respectively, the dropout can apply to the RNN, which is called a variational dropout BID4 . This paper proposes a dense RNN that has both of feedforward and recurrent depths. The stacked RNN increases the complexity by increasing feedforward depth. NARX-RNN and HO-RNN increase the complexity by increasing recurrent depth. The model with the feedforward depth can be combined with the model with the recurrent depth, as the feedforward depth and recurrent depth have an orthogonal relationship. Gated feedback RNN has the fully connection between two consecutive timesteps. As the connection of gated feedback is not overlapped with the model with orthogonal depths, all three features, adding feedforward depth, recurrent depth, and gated feedback, can be modeled jointly . With the three features, we propose the attention gate, which controls the flows from each state so that it enhances the overall performance.The contributions of this paper are summarized: 1) dense RNN that is aggregated model with feedforward depth, recurrent depth and gated feedback function, 2) extension of the variational dropout to the dense RNN. This paper proposed dense RNN, which has fully connections from each hidden state to multiple preceding hidden states of all layers directly. Each previous hidden state has its attention gate that controls the amount of information flows. To evaluate the effect of dense connections, we used Penn Treebank corpus (PTB). The result of dense connection was confirmed by varying the recurrent depth with the attention gate. The dense connections with the attention gate made the model's perplexity less than conventional RNN.
Dense RNN that has fully connections from each hidden state to multiple preceding hidden states of all layers directly. Proposes a new RNN architecture that models long-term dependencies better, can learn multiscale representation of sequential data, and sidestep the gradients problem by using parametrized gating units. This paper proposes a fully connected dense RNN architecture with gated connections to every layer and preceding layer connections, and it's results on PTB charcter-level modelling task.