3VL: using Trees to teach Vision & Language models compositional
  concepts
Nir Yellinek Leonid Karlinsky Raja Giryes
Vision-Language models (VLMs) have proved effective at aligning image and
text representations, producing superior zero-shot results when transferred to
many downstream tasks. However, these representations suffer some key
shortcomings in Compositional Language Concepts (CLC) understanding such as
recognizing objects' attributes, states, and relations between different
objects. Moreover, VLMs typically have poor interpretability, making it
challenging to debug and mitigate compositional-understanding failures. In this
work, we introduce the Tree-augmented Vision-Language (3VL) model architecture
and training technique accompanied by our proposed Anchor inference method and
Differential Relevance (DiRe) interpretability tool. By expanding the text of
an arbitrary image-text pair into a hierarchical tree structure using language
analysis tools, 3VL allows inducing this structure into the visual
representation learned by the model, enhancing its interpretability and
compositional reasoning. Additionally, we show how Anchor, a simple technique
for text unification, can be employed to filter nuisance factors while
increasing CLC understanding performance, e.g., on the fundamental VL-Checklist
benchmark. We also exhibit how DiRe, which performs a differential comparison
between VLM relevancy maps, enables us to generate compelling visualizations of
the reasons for a model's success or failure.