A Theoretical Result on the Inductive Bias of RNN Language Models
Anej Svete Robin Shing Moon Chan Ryan Cotterell
Recent work by Hewitt et al. (2020) provides a possible interpretation of the
empirical success of recurrent neural networks (RNNs) as language models (LMs).
  It shows that RNNs can efficiently represent bounded hierarchical structures
that are prevalent in human language.
  This suggests that RNNs' success might be linked to their ability to model
hierarchy.
  However, a closer inspection of Hewitt et al.'s (2020) construction shows
that it is not limited to hierarchical LMs, posing the question of what
\emph{other classes} of LMs can be efficiently represented by RNNs.
  To this end, we generalize their construction to show that RNNs can
efficiently represent a larger class of LMs: Those that can be represented by a
pushdown automaton with a bounded stack and a generalized stack update
function.
  This is analogous to an automaton that keeps a memory of a fixed number of
symbols and updates the memory with a simple update mechanism.
  Altogether, the efficiency in representing a diverse class of
non-hierarchical LMs posits a lack of concrete cognitive and
human-language-centered inductive biases in RNNs.