Modeling rapid language learning by distilling Bayesian priors into
  artificial neural networks
R. Thomas McCoy Thomas L. Griffiths
Humans can learn languages from remarkably little experience. Developing
computational models that explain this ability has been a major challenge in
cognitive science. Bayesian models that build in strong inductive biases -
factors that guide generalization - have been successful at explaining how
humans might generalize from few examples in controlled settings but are
usually too restrictive to be tractably applied to more naturalistic data. By
contrast, neural networks have flexible representations that allow them to
learn well from naturalistic data but require many more examples than humans
receive. We show that learning from limited naturalistic data is possible with
an approach that combines the strong inductive biases of a Bayesian model with
the flexible representations of a neural network. This approach works by
distilling a Bayesian model's biases into a neural network. Like a Bayesian
model, the resulting system can learn formal linguistic patterns from a small
number of examples. Like a neural network, it can also learn aspects of English
syntax from a corpus of natural language - and it outperforms a standard neural
network at acquiring the linguistic phenomena of recursion and priming.
Bridging the divide between Bayesian models and neural networks makes it
possible to handle a broader range of learning scenarios than either approach
can handle on its own.