Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate
Pengfei Sun Jibin Wu Malu Zhang Paul Devos Dick Botteldooren
Recurrent Neural Networks (RNNs) are renowned for their adeptness in modeling
temporal dependencies, a trait that has driven their widespread adoption for
sequential data processing. Nevertheless, vanilla RNNs are confronted with the
well-known issue of gradient vanishing and exploding, posing a significant
challenge for learning and establishing long-range dependencies. Additionally,
gated RNNs tend to be over-parameterized, resulting in poor network
generalization. To address these challenges, we propose a novel Delayed Memory
Unit (DMU) in this paper, wherein a delay line structure, coupled with delay
gates, is introduced to facilitate temporal interaction and temporal credit
assignment, so as to enhance the temporal modeling capabilities of vanilla
RNNs. Particularly, the DMU is designed to directly distribute the input
information to the optimal time instant in the future, rather than aggregating
and redistributing it over time through intricate network dynamics. Our
proposed DMU demonstrates superior temporal modeling capabilities across a
broad range of sequential modeling tasks, utilizing considerably fewer
parameters than other state-of-the-art gated RNN models in applications such as
speech recognition, radar gesture recognition, ECG waveform segmentation, and
permuted sequential image classification.