Empirical Analysis of the Inductive Bias of Recurrent Neural Networks by
  Discrete Fourier Transform of Output Sequences
Taiga Ishii Ryo Ueda Yusuke Miyao
A unique feature of Recurrent Neural Networks (RNNs) is that it incrementally
processes input sequences. In this research, we aim to uncover the inherent
generalization properties, i.e., inductive bias, of RNNs with respect to how
frequently RNNs switch the outputs through time steps in the sequence
classification task, which we call output sequence frequency. Previous work
analyzed inductive bias by training models with a few synthetic data and
comparing the model's generalization with candidate generalization patterns.
However, when examining the output sequence frequency, previous methods cannot
be directly applied since enumerating candidate patterns is computationally
difficult for longer sequences. To this end, we propose to directly calculate
the output sequence frequency for each model by regarding the outputs of the
model as discrete-time signals and applying frequency domain analysis.
Experimental results showed that Long Short-Term Memory (LSTM) and Gated
Recurrent Unit (GRU) have an inductive bias towards lower-frequency patterns,
while Elman RNN tends to learn patterns in which the output changes at high
frequencies. We also found that the inductive bias of LSTM and GRU varies with
the number of layers and the size of hidden layers.