Recurrent Neural Networks architectures excel at processing sequences by
 modelling dependencies over different timescales. The recently introduced
 Recurrent Weighted Average (RWA) unit captures long term dependencies
 far better than an LSTM on several challenging tasks. The RWA achieves
 this by applying attention to each input and computing a weighted average
 over the full history of its computations. Unfortunately, the RWA cannot
 change the attention it has assigned to previous timesteps, and so struggles
 with carrying out consecutive tasks or tasks with changing requirements.
 We present the Recurrent Discounted Attention (RDA) unit that builds on
 the RWA by additionally allowing the discounting of the past.
 We empirically compare our model to RWA, LSTM and GRU units on
 several challenging tasks. On tasks with a single output the RWA, RDA and
 GRU units learn much quicker than the LSTM and with better performance.
 On the multiple sequence copy task our RDA unit learns the task three
 times as quickly as the LSTM or GRU units while the RWA fails to learn at
 all. On the Wikipedia character prediction task the LSTM performs best
 but it followed closely by our RDA unit. Overall our RDA unit performs
 well and is sample efficient on a large variety of sequence tasks. Many types of information such as language, music and video can be represented as sequential data. Sequential data often contains related information separated by many timesteps, for instance a poem may start and end with the same line, a scenario which we call long term dependencies. Long term dependencies are difficult to model as we must retain information from the whole sequence and this increases the complexity of the model. A class of model capable of capturing long term dependencies are Recurrent Neural Networks (RNNs). A specific RNN architecture, known as Long Short-Term Memory (LSTM) BID13 , is the benchmark against which other RNNs are compared. LSTMs have been shown to learn many difficult sequential tasks effectively. They store information from the past within a hidden state that is combined with the latest input at each timestep. This hidden state can carry information right from the beginning of the input sequence, which allows long term dependencies to be captured. However, the hidden state tends to focus on the more recent past and while this mostly works well, in tasks requiring equal weighting between old and new information LSTMs can fail to learn.A technique for accessing information from anywhere in the input sequence is known as attention. The attention mechanism was introduced to RNNs by BID2 for neural machine translation. The text to translate is first encoded by a bidirectional-RNN producing a new sequence of encoded state. Different locations within the encoded state are focused on by multiplying each of them by an attention matrix and calculating the weighted average. This attention is calculated for each translated word. Computing the attention matrix for each encoded state and translated word combination provides a great deal of flexibility in choosing where in the sequence to attend to, but the cost of computing these matrices grows as a square of the number of words to translate. This cost limits this method to short sequences, typically only single sentences are processed at a time.The Recurrent Weighted Average (RWA) unit, recently introduced by BID17 , can apply attention to sequences of any length. It does this by only computing the attention for each input once and computing the weighted average by maintaining a running average up to the current timestep. Their experiments show that the RWA performs very well on tasks where information is needed from any point in the input sequence. Unfortunately, as it cannot change the attention it assigns to previous timesteps, it performs poorly when asked to carry out multiple tasks within the same sequence, or when asked to predict the next character in a sample of text, a task in which new information is more important than old.We introduce the Recurrent Discounted Attention (RDA) unit, which extends the RWA by allowing it to discount the attention applied to previous timesteps. As this adjustment is applied to all previous timesteps at once, it continues to be efficient. It performs very well both at tasks requiring equal weighting over all information seen and at tasks in which new information is more important than old.The main contributions of this paper are as follows:1. We analyse the Recurrent Weighted Average unit and show that it cannot output certain simple sequences.2. We propose the Recurrent Discounted Attention unit that extends the Recurrent Weighted Average by allowing it to discount the past.3. We run extensive experiments on the RWA, RDA, LSTM and GRU units and show that the RWA, RDA and GRU units are well suited to tasks with a single output, the RDA performs best on the multiple sequence copy task while the LSTM unit performs better on the Hutter Prize Wikipedia dataset.Our paper is setout as follows: we present the analysis of the RWA (sections 3 and 4) and propose the RDA (section 5). The experimental results (section 6), discussion (section 7) and conclusion follow (section 8). We analysed the Recurrent Weighted Average (RWA) unit and identified its weakness as the inability to forget the past. By adding this ability to forget the past we arrived at the Recurrent Discounted Attention (RDA). We implemented several varieties of the RDA and compared them to the RWA, LSTM and GRU units on several different tasks. We showed that in almost all cases the RDA should be used in preference to the RWA and is a flexible RNN unit that can perform well on all types of tasks.We also determined which types of tasks were more suited to each different RNN unit. For tasks involving a single output the RWA, RDA and GRU units performed best, for the multiple sequence copy task the RDA performed best, while on the Wikipedia character prediction task the LSTM unit performed best. We recommend taking these results into account when choosing a unit for real world applications.
We introduce the Recurrent Discounted Unit which applies attention to any length sequence in linear time This paper proposes the Recurrent Discounted Attention (RDA), an extension to Recurrent Weighted Average (RWA) by adding a discount factor. Extends the recurrent weight average to overcome the limitation of the original method while maintaining its advantage and proposes the method of using Elman nets as the base RNN