In this work, we focus on the problem of grounding language by training an agent
 to follow a set of natural language instructions and navigate to a target object
 in a 2D grid environment. The agent receives visual information through raw
 pixels and a natural language instruction telling what task needs to be achieved.
 Other than these two sources of information, our model does not have any prior
 information of both the visual and textual modalities and is end-to-end trainable.
 We develop an attention mechanism for multi-modal fusion of visual and textual
 modalities that allows the agent to learn to complete the navigation tasks and also
 achieve language grounding. Our experimental results show that our attention
 mechanism outperforms the existing multi-modal fusion mechanisms proposed in
 order to solve the above mentioned navigation task. We demonstrate through the
 visualization of attention weights that our model learns to correlate attributes of
 the object referred in the instruction with visual representations and also show
 that the learnt textual representations are semantically meaningful as they follow
 vector arithmetic and are also consistent enough to induce translation between instructions
 in different natural languages. We also show that our model generalizes
 effectively to unseen scenarios and exhibit zero-shot generalization capabilities.
 In order to simulate the above described challenges, we introduce a new 2D environment
 for an agent to jointly learn visual and textual modalities Figure 1: The agent (blue in color) should learn to read the instruction and navigate to green apple.Understanding of natural language instructions is an important aspect of an Artificial Intelligence (AI) system. In order to successfully accomplish tasks specified by natural language instructions, an agent has to extract representations of language that are semantically meaningful and ground it in perceptual elements and actions in the environment.Humans have the ability to understand the true essence of the words and thus they can easily decipher sentences even if it contains some new combination of words. It is not unreasonable to expect the same from an AI agent. The information extracted by agent from the language should be such that it corresponds to the true meaning of the word so that it enables the agent to generalize to even unseen combinations of words. For instance, when given sufficient information about the words such as 'green' and 'bag', it should automatically figure out as to what 'green bag' essentially means.Consider a task in which an agent has to learn to navigate to a target object in a 2D grid environment as shown in figure 1. The environment consists of many objects with different attributes (in our case: green apple, red apple, blue sofa, green sofa, an orange fruit, red car) and multiple obstacles. The agent receives visual information through raw pixels and an instruction telling what task needs to be achieved. The challenges that the agent has to tackle here are manyfold: a) the agent has to develop the capability to recognize various objects, b) have some memory of objects seen in previous states while exploring the environment as the objects may occlude each other and/or may not be present in the agent's field of view c) ground the instruction in visual elements and actions in the environment and d) learn a policy to navigate to the target object by avoiding the obstacles and other non-target objects.We tackle this problem by proposing an end-to-end trainable architecture that creates a combined representation of the image observed by the agent and the instruction it receives. Our model does not have any prior information of both the visual and textual modalities. We develop an attention mechanism for multimodal fusion of visual and textual modalities. Our experimental results show that our attention mechanism outperforms the existing multimodal fusion mechanisms proposed in order to solve the above mentioned task. We demonstrate through the visualization of attention weights that our model learns to correlate attributes of the object referred in the instruction with visual representations and also show that the learnt textual representations are semantically meaningful as they follow vector arithmetic and are also consistent enough to induce translation between instructions in different natural languages. We also show that our model generalizes effectively to unseen scenarios and exhibit zero-shot (ZS) generalization capabilities. In order to simulate the above described challenges, we introduce a new 2D environment for an agent to jointly learn visual and textual modalities. Our 2D environment is also thread compatible. Finally, in order to enable the reproducibility of our research through participation in BID21 and foster further research in this direction, we open source our environment as well as the code and models that we developed. In the paper we presented an attention based simple architecture to achieve grounding of natural language sentences via reinforcement learning. We show that retaining just the representation obtained after multimodal fusion phase (i.e. multiple attention maps) and discarding the visual features helps the agent achieve its goals. We justify this claim by visualization of the attention maps which reveal that they contain sufficient information needed for the agent to find the optimal policy. Through vector arithmetic, we also show that the embeddings learnt by the agent indeed make sense. In order to encourage the research in this direction we have also open sourced our environment as well as the code and models developed.Our environment is capable of supporting rich set of natural language instructions and it's highly flexible. As a future work, we would like to increase the complexity of both the types of sentences generated as well as the environment dynamics by introducing moving objects. We also plan to take forward our approach to a 3D environment to test how well does it extend there. To test if our attention mechanism could scale to 3D environments, we applied our fusion mechanism to the Vizdoom based environment and compared our results with who have recently open sourced their code Chaplot. We replaced their fusion mechanism with our attention based fusion mechanism and the results for the easy, medium and also hard scenario are shown in FIG11 . We have also compared our accuracy values with the results mentioned by the authors in their paper, in table 3, whereby we show significant performance improvement during the test phase including for the zeros shot instructions. The experiments are still running for the hard difficulty case but the results show that our approach converges much faster than the original fusion mechanism used by them. We see that our mechanism thus scales to both 2D as well as 3D environments outperforming other baselines in both the scenarios. The experiments were conducted on the same set of hardware thus ensuring a fair comparison between the plots.The plot shown here differs from the one shown in the paper ) because of a possible difference in hardwares used. We will update the plots as soon as all the experiments have been completed.
Attention based architecture for language grounding via reinforcement learning in a new customizable 2D grid environment   The paper tackles the problem of navigation given an instruction and proposes an approach to combine textual and visual information via an attention mechanism This paper considers the problem of following natural language instructions given a first-person view of an a priori unknown environment, and proposes a neural architecture method. Studies the problem of navigating to a target object in a 2D grid environment by following given natural language description and receiving visual information as raw pixels.