Analyzing the Roles of Language and Vision in Learning from Limited Data
Allison Chen Ilia Sucholutsky Olga Russakovsky Thomas L. Griffiths
Does language help make sense of the visual world? How important is it to
actually see the world rather than having it described with words? These basic
questions about the nature of intelligence have been difficult to answer
because we only had one example of an intelligent system -- humans -- and
limited access to cases that isolated language or vision. However, the
development of sophisticated Vision-Language Models (VLMs) by artificial
intelligence researchers offers us new opportunities to explore the
contributions that language and vision make to learning about the world. We
ablate components from the cognitive architecture of these models to identify
their contributions to learning new tasks from limited data. We find that a
language model leveraging all components recovers a majority of a VLM's
performance, despite its lack of visual input, and that language seems to allow
this by providing access to prior knowledge and reasoning.