Neural network-based systems can now learn to locate the referents of words and phrases in images, answer questions about visual scenes, and even execute symbolic instructions as first-person actors in partially-observable worlds. To achieve this so-called grounded language learning, models must overcome certain well-studied learning challenges that are also fundamental to infants learning their first words. While it is notable that models with no meaningful prior knowledge overcome these learning obstacles, AI researchers and practitioners currently lack a clear understanding of exactly how they do so. Here we address this question as a way of achieving a clearer general understanding of grounded language learning, both to inform future research and to improve confidence in model predictions. For maximum control and generality, we focus on a simple neural network-based language learning agent trained via policy-gradient methods to interpret synthetic linguistic instructions in a simulated 3D world. We apply experimental paradigms from developmental psychology to this agent, exploring the conditions under which established human biases and learning effects emerge. We further propose a novel way to visualise and analyse semantic representation in grounded language learning agents that yields a plausible computational account of the observed effects. The learning challenge faced by children acquiring their first words has long fascinated cognitive scientists and philosophers BID34 BID5 . To start making sense of language, an infant must induce structure in a constant stream of continuous visual input, slowly reconcile this structure with consistencies in the available linguistic observations, store this knowledge in memory, and apply it to inform decisions about how best to respond.Many neural network models also overcome a learning task that is -to varying degrees -analogous to early human word learning. Image classification tasks such as the ImageNet Challenge BID8 ) require models to induce discrete semantic classes, in many cases aligned to words, from unstructured pixel representations of large quantities of photographs BID20 . Visual question answering (VQA) systems BID0 BID41 BID42 ) must reconcile raw images with (arbitrary-length) sequences of symbols, in the form of natural language questions, in order to predict lexical or phrasal answers. Recently, situated language learning agents have been developed that learn to understand sequences of linguistic symbols not only in terms of the contemporaneous raw visual input, but also in terms of past visual input and the actions required to execute an appropriate motor response BID29 BID6 BID15 BID26 . The most advanced such agents learn to execute a range of phrasal and multi-task instructions, such as find the green object in the red room, pick up the pencil in the third room on the right or go to the small green torch, in a continous, simulated 3D world. To solve these tasks, an agent must execute sequences of hundreds of fine-grained actions, conditioned on the available sequence of language symbols and active (first-person) visual perception of the surroundings. Importantly, the knowledge acquired by such agents while mastering these tasks also permits the interpretation of familiar language in entirely novel surroundings, and the execution of novel instructions composed of combinations of familiar words BID6 BID15 .The potential impact of situated linguistic agents, VQA models and other grounded language learning systems is vast, as a basis for human users to interact with situated learning applications such as self-driving cars and domestic robotic tools. However , our understanding of how these agents learn and behave is limited. The challenges of interpreting the factors or reasoning behind the decisions and predictions of neural networks are well known. Indeed, a concerted body of research in both computer vision BID44 BID38 BID43 and natural language processing BID23 BID39 has focused on addressing this uncertainty. As grounded language learning agents become more prevalent, then, understanding their learning dynamics, representation and decision-making will become increasingly important, both to inform future research and to build confidence in users who interact with such models.We therefore aim to establish a better understanding of neural network-based models of grounded language learning, noting the parallels with research in neuroscience and psychology that aims to understand human language acquisition. Extending the approach of BID35 , we adapt various experimental techniques initially developed by experimental psychologists BID21 BID24 BID17 BID7 . In line with typical experiments on humans, our experimental simulations are conducted in a highly controlled environment: a simulated 3D world with a limited set of objects and properties, and corresponding unambiguous, symbolic linguistic stimuli (Figure 1) . However, the simplicity and generality of our architecture and the form of the inputs to the model (continuous visual plus symbolic linguistic) make the proposed methods and approach directly applicable to VQA and other tasks that combine linguistic and visual data. Using these methods, we explore how the training environment of our agent affects its learning outcomes and speed, measure the generality and robustness of its understanding of certain fundamental linguistic concepts, and test for biases in the decisions it takes once trained. Further, by applying layerwise attention , a novel tool for visualising computation in grounded language learning models, we obtain a plausible algorithmic account of some of the effects in terms of representation and processing. Our principal findings about this canonical grounded language learning architecture are the following:Shape / colour biases When the agent is trained on an equal number of shape and colour words, it develops a propensity to extend labels for ambiguous new words according to colour rather than shape (color bias). A human-like bias towards shapes can be induced in the agent, but only if it experiences many more shape terms than colour terms during training.The problem of learning negation The agent learns to execute negated instructions, but if trained on small amounts of data it tends to represent negation in an ad hoc way that does not generalise.Curriculum effects for vocabulary growth The agent learns words more quickly if the range of words to which it is exposed is limited at first and expanded gradually as its vocabulary develops.Semantic processing and representation differences The agent learns words of different semantic classes at different speeds and represents them with features that require different degrees of visual processing depth (or abstraction) to compute.Before describing the experiments that reveal these effects, we briefly outline details of the environment and agent used for the simulations. Models that are capable of grounded language learning promise to significantly advance the ways in which humans and intelligent technology can interact. In this study, we have explored how a situated language learning agent built from canonical neural-network components overcomes the challenge of early language learning. We measured the behaviour exhibited once the first words and simple phrases are acquired, tested factors that speed up this learning, explored aspects of language that pose particular problems and presented a technique, layerwise attention, for better understanding semantic and visual processing in such agents.The application of experimental paradigms from cognitive psychology to better understand deep neural nets was proposed by BID35 , who observed that convolutional architectures exhibit a shape bias when trained on the ImageNet Challenge data. The ability to control precisely both training and test stimuli in our simulated environment allowed us to isolate this effect as deriving from the training data, and indeed to reach the opposite conclusion about the architecture itself. This study also goes beyond that of BID35 in exploring more abstract linguistic operations (negation, abstract category terms) and studying curriculum effects on the dynamics of word learning. Further, we complement these behavioural observations with computatoinal analysis of representation and processing, via layerwise attention.While the control and precision afforded by the simulated environment in the present study has made these analyses and conclusions possible, in future, as our understanding of language learning agents develops, it will be essential to verify conclusions on agents trained on more naturalistic data. At first, this might involve curated sets of images, videos and naturally-occurring text etc, and, ultimately, experiments on robots trained to communicate about perceptible surroundings with human interlocutors. In a world with agents capable of learning such advanced linguistic behaviour, it would certainly be more challenging, but also even more crucial, to understand not just what they can do, but also how they learn to do it. 6 SUPPLEMENTARY MATERIAL
Analysing and understanding how neural network agents learn to understand simple grounded language The authors connect psychological experimental methods to understanding how the black box of deep learning methods solves problems. This paper presents an analysis of the agents who learn grounded language through reinforcement learning in a simple environment that combines verbal instruction with visual information