Syntax-Guided Transformers: Elevating Compositional Generalization and
  Grounding in Multimodal Environments
Danial Kamali Parisa Kordjamshidi
Compositional generalization, the ability of intelligent models to
extrapolate understanding of components to novel compositions, is a fundamental
yet challenging facet in AI research, especially within multimodal
environments. In this work, we address this challenge by exploiting the
syntactic structure of language to boost compositional generalization. This
paper elevates the importance of syntactic grounding, particularly through
attention masking techniques derived from text input parsing. We introduce and
evaluate the merits of using syntactic information in the multimodal grounding
problem. Our results on grounded compositional generalization underscore the
positive impact of dependency parsing across diverse tasks when utilized with
Weight Sharing across the Transformer encoder. The results push the
state-of-the-art in multimodal grounding and parameter-efficient modeling and
provide insights for future research.