Development of Compositionality and Generalization through Interactive
  Learning of Language and Action of Robots
Prasanna Vijayaraghavan Jeffrey Frederic Queisser Sergio Verduzco Flores Jun Tani
Humans excel at applying learned behavior to unlearned situations. A crucial
component of this generalization behavior is our ability to compose/decompose a
whole into reusable parts, an attribute known as compositionality. One of the
fundamental questions in robotics concerns this characteristic. "How can
linguistic compositionality be developed concomitantly with sensorimotor skills
through associative learning, particularly when individuals only learn partial
linguistic compositions and their corresponding sensorimotor patterns?" To
address this question, we propose a brain-inspired neural network model that
integrates vision, proprioception, and language into a framework of predictive
coding and active inference, based on the free-energy principle. The
effectiveness and capabilities of this model were assessed through various
simulation experiments conducted with a robot arm. Our results show that
generalization in learning to unlearned verb-noun compositions, is
significantly enhanced when training variations of task composition are
increased. We attribute this to self-organized compositional structures in
linguistic latent state space being influenced significantly by sensorimotor
learning. Ablation studies show that visual attention and working memory are
essential to accurately generate visuo-motor sequences to achieve
linguistically represented goals. These insights advance our understanding of
mechanisms underlying development of compositionality through interactions of
linguistic and sensorimotor experience.