RelationVLM: Making Large Vision-Language Models Understand Visual
  Relations
Zhipeng Huang Zhizheng Zhang Zheng-Jun Zha Yan Lu Baining Guo
The development of Large Vision-Language Models (LVLMs) is striving to catch
up with the success of Large Language Models (LLMs), yet it faces more
challenges to be resolved. Very recent works enable LVLMs to localize
object-level visual contents and ground text to them. Nonetheless, current
LVLMs still struggle to precisely understand visual relations due to the lack
of relevant data. In this work, we present RelationVLM, a large vision-language
model capable of comprehending various levels and types of relations whether
across multiple images or within a video. Specifically, we devise a multi-stage
relation-aware training scheme and a series of corresponding data configuration
strategies to bestow RelationVLM with the capabilities of understanding
semantic relations, temporal associations and geometric transforms. Extensive
case studies and quantitative evaluations show RelationVLM has strong
capability in understanding such relations and emerges impressive in-context
capability of reasoning from few-shot examples by comparison. This work fosters
the advancements of LVLMs by enabling them to support a wider range of
downstream applications toward artificial general intelligence.