Humans can understand and produce new utterances effortlessly, thanks to their systematic compositional skills. Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax." In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can generalize well when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task. However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, supporting the conjecture that lack of systematicity is an important factor explaining why neural networks need very large training sets. Human language and thought are characterized by systematic compositionality, the algebraic capacity to understand and produce a potentially infinite number of novel combinations from known components. For example, if a person knows the meaning and usage of words such as "twice," "and," and "again," once she learns a new verb such as "to dax" she can immediately understand or produce instructions such as "dax twice and then dax again." This type of compositionality is central to the human ability to make strong generalizations from very limited data. In a set of influential and controversial papers, Jerry Fodor and other researchers have argued that neural networks are not plausible models of the mind because they are associative devices that cannot capture systematic compositionality BID11 BID22 BID10 BID23 Calvo & Symons, 2014, a.o.) .In the last few years, neural network research has made astounding progress in practical domains where success crucially depends on the generalization capabilities of a system. Perhaps most strikingly, end-to-end recurrent neural networks currently dominate the state-of-the-art in machine translation BID2 BID30 . 1 Since the overwhelming majority of sentences or even word sequences in a language only occur once, even in a large corpus BID1 , this points to strong generalization abilities. Still, it is commonly observed that neural networks are extremely sample inefficient, requiring very large training sets, which suggests they may lack the same algebraic compositionality that humans exploit, and they might only be sensitive to broad patterns over lots of accumulated statistics BID21 .In this paper, we introduce a grounded navigation environment where the learner must translate commands given in a limited form of natural language into a sequence of actions. This problem is naturally framed as a sequence-to-sequence task, and, due to its simplicity, it is ideal to study the systematic generalization capabilities of computational systems to novel examples in a controlled setup. We thus use it to test a wide range of modern recurrent network architectures in terms of their compositional skills. Our results suggest that, although standard architectures such as LSTMs with attention BID0 do generalize when novel examples feature a mixture of constructions that have been observed in training, the models are catastrophically affected by systematic differences between training and test sentences, of the sort that would be trivial for an agent equipped with an "algebraic mind" BID23 . In the thirty years since the inception of the systematicity debate, many authors on both sides have tested the ability of neural networks to solve tasks requiring compositional generalization, with mixed results (e.g., BID6 BID22 BID25 BID5 BID27 BID29 BID3 BID12 . However, to the best of our knowledge, ours is the first study testing systematicity in modern seq2seq models, and our results confirm the mixed picture. On the one hand, standard recurrent models can reach very high zero-shot accuracy from relatively few training examples, as long as the latter are generally representative of the test data (Experiment 1). However, the same networks fail spectacularly when there are systematic differences between training and testing. Crucially, the training data of the relevant experiments provide enough evidence to learn composition rules affording the correct generalizations. In Experiment 2, the training data contain examples of all modifiers and connectives that are needed at test time for producing longer action sequences. In Experiment 3, the usage of modifiers and connectives is illustrated at training time by their application to some primitive commands, and, at test time, the model should apply them to a new command it encountered in isolation during training. Nonetheless, this evidence was not sufficient for each of the networks we tested. Generalization only occurs when the networks are also exposed to the target command (or the corresponding action) in a fair number of composed contexts during training.Given the astounding successes of seq2seq models in challenging tasks such as machine translation, one might argue that failure to generalize by systematic composition indicates that neural networks are poor models of some aspects of human cognition, but it is of little practical import. However, systematicity is an extremely efficient way to generalize. Once a person learns the new English adjective "daxy", he or she can immediately produce and understand an infinity of sentences containing it. The SCAN experiments and a proof-of-concept machine translation experiment (Experiment 4) suggest that this ability is still beyond the grasp of state-of-the-art networks, likely contributing to their striking sample-inefficiency. These results give us hope that a model capable of systematic compositionality could greatly benefit machine translation and other applications.A natural way of achieving stronger compositionality is through learning more structured representations. Recently, neural networks with external memories have shown promise for extracting algorithm-like representations from input/output examples BID19 BID15 ; for instance, these networks can outperform standard RNNs on generalizing to longer sequences. Future work will explore these approaches on SCAN and other tests of zero-shot compositional generalization. Ultimately, we see systematic compositionality as key both to developing more powerful algorithms and to enriching our computational understanding of the human mind.
Using a simple language-driven navigation task, we study the compositional capabilities of modern seq2seq recurrent networks. 
This paper focuses on the zero-shot learning compositional capabilities of modern sequence-to-sequence RNNs and  exposes the short-comings of current seq2seq RNN architectures. The paper analyzes the composition abilities of RNNs, specifically, the generalization ability of RNNs on random subset of SCAN commands, on longer SCAN commands, and of composition over primitive commands.  The authors introduce a new dataset that facilitates the analysis of a Seq2Seq learning case