World-to-Words: Grounded Open Vocabulary Acquisition through Fast
  Mapping in Vision-Language Models
Ziqiao Ma Jiayi Pan Joyce Chai
The ability to connect language units to their referents in the physical
world, referred to as grounding, is crucial to learning and understanding
grounded meanings of words. While humans demonstrate fast mapping in new word
learning, it remains unclear whether modern vision-language models can truly
represent language with their grounded meanings and how grounding may further
bootstrap new word learning. To this end, we introduce Grounded Open Vocabulary
Acquisition (GOVA) to examine grounding and bootstrapping in open-world
language learning. As an initial attempt, we propose object-oriented BERT
(OctoBERT), a novel visually-grounded language model by pre-training on
image-text pairs highlighting grounding as an objective. Through extensive
experiments and analysis, we demonstrate that OctoBERT is a more coherent and
fast grounded word learner, and that the grounding ability acquired during
pre-training helps the model to learn unseen words more rapidly and robustly.
Our code is available at https://github.com/sled-group/world-to-words