Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs
  Collaborated Reasoning
Hang Zhang Wenxiao Zhang Haoxuan Qu Jun Liu
Human-centered dynamic scene understanding plays a pivotal role in enhancing
the capability of robotic and autonomous systems, in which Video-based
Human-Object Interaction (V-HOI) detection is a crucial task in semantic scene
understanding, aimed at comprehensively understanding HOI relationships within
a video to benefit the behavioral decisions of mobile robots and autonomous
driving systems. Although previous V-HOI detection models have made significant
strides in accurate detection on specific datasets, they still lack the general
reasoning ability like human beings to effectively induce HOI relationships. In
this study, we propose V-HOI Multi-LLMs Collaborated Reasoning (V-HOI MLCR), a
novel framework consisting of a series of plug-and-play modules that could
facilitate the performance of current V-HOI detection models by leveraging the
strong reasoning ability of different off-the-shelf pre-trained large language
models (LLMs). We design a two-stage collaboration system of different LLMs for
the V-HOI task. Specifically, in the first stage, we design a Cross-Agents
Reasoning scheme to leverage the LLM conduct reasoning from different aspects.
In the second stage, we perform Multi-LLMs Debate to get the final reasoning
answer based on the different knowledge in different LLMs. Additionally, we
devise an auxiliary training strategy that utilizes CLIP, a large
vision-language model to enhance the base V-HOI models' discriminative ability
to better cooperate with LLMs. We validate the superiority of our design by
demonstrating its effectiveness in improving the prediction accuracy of the
base V-HOI model via reasoning from multiple perspectives.