Positional Encoding Helps Recurrent Neural Networks Handle a Large
  Vocabulary
Takashi Morita
This study discusses the effects of positional encoding on recurrent neural
networks (RNNs) utilizing synthetic benchmarks. Positional encoding
"time-stamps" data points in time series and complements the capabilities of
Transformer neural networks, which lack an inherent mechanism for representing
the data order. By contrast, RNNs can encode the temporal information of data
points on their own, rendering their use of positional encoding seemingly
"redundant". Nonetheless, empirical investigations reveal the effectiveness of
positional encoding even when coupled with RNNs, specifically for handling a
large vocabulary that yields diverse observations. These findings pave the way
for a new line of research on RNNs, concerning the combination of input-driven
and autonomous time representation. Additionally, biological implications of
the computational/simulational results are discussed, in the light of the
affinity between the sinusoidal implementation of positional encoding and
neural oscillations in biological brains.