BRAVE: Broadening the visual encoding of vision-language models
Oğuzhan Fatih Kar Alessio Tonioni Petra Poklukar Achin Kulshrestha Amir Zamir Federico Tombari
Vision-language models (VLMs) are typically composed of a vision encoder,
e.g. CLIP, and a language model (LM) that interprets the encoded features to
solve downstream tasks. Despite remarkable progress, VLMs are subject to
several shortcomings due to the limited capabilities of vision encoders, e.g.
"blindness" to certain image features, visual hallucination, etc. To address
these issues, we study broadening the visual encoding capabilities of VLMs. We
first comprehensively benchmark several vision encoders with different
inductive biases for solving VLM tasks. We observe that there is no single
encoding configuration that consistently achieves top performance across
different tasks, and encoders with different biases can perform surprisingly
similarly. Motivated by this, we introduce a method, named BRAVE, that
consolidates features from multiple frozen encoders into a more versatile
representation that can be directly fed as the input to a frozen LM. BRAVE
achieves state-of-the-art performance on a broad range of captioning and VQA
benchmarks and significantly reduces the aforementioned issues of VLMs, while
requiring a smaller number of trainable parameters than existing methods and
having a more compressed representation. Our results highlight the potential of
incorporating different visual biases for a more broad and contextualized
visual understanding of VLMs.