Hierarchically Gated Recurrent Neural Network for Sequence Modeling
Zhen Qin Songlin Yang Yiran Zhong
Transformers have surpassed RNNs in popularity due to their superior
abilities in parallel training and long-term dependency modeling. Recently,
there has been a renewed interest in using linear RNNs for efficient sequence
modeling. These linear RNNs often employ gating mechanisms in the output of the
linear recurrence layer while ignoring the significance of using forget gates
within the recurrence. In this paper, we propose a gated linear RNN model
dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes
forget gates that are lower bounded by a learnable value. The lower bound
increases monotonically when moving up layers. This allows the upper layers to
model long-term dependencies and the lower layers to model more local,
short-term dependencies. Experiments on language modeling, image
classification, and long-range arena benchmarks showcase the efficiency and
effectiveness of our proposed model. The source code is available at
https://github.com/OpenNLPLab/HGRN.