$\text{R}^2$-Bench: Benchmarking the Robustness of Referring Perception
  Models under Perturbations
Xiang Li Kai Qiu Jinglu Wang Xiaohao Xu Rita Singh Kashu Yamazak Hao Chen Xiaonan Huang Bhiksha Raj
Referring perception, which aims at grounding visual objects with multimodal
referring guidance, is essential for bridging the gap between humans, who
provide instructions, and the environment where intelligent systems perceive.
Despite progress in this field, the robustness of referring perception models
(RPMs) against disruptive perturbations is not well explored. This work
thoroughly assesses the resilience of RPMs against various perturbations in
both general and specific contexts. Recognizing the complex nature of referring
perception tasks, we present a comprehensive taxonomy of perturbations, and
then develop a versatile toolbox for synthesizing and evaluating the effects of
composite disturbances. Employing this toolbox, we construct
$\text{R}^2$-Bench, a benchmark for assessing the Robustness of Referring
perception models under noisy conditions across five key tasks. Moreover, we
propose the $\text{R}^2$-Agent, an LLM-based agent that simplifies and
automates model evaluation via natural language instructions. Our investigation
uncovers the vulnerabilities of current RPMs to various perturbations and
provides tools for assessing model robustness, potentially promoting the safe
and resilient integration of intelligent systems into complex real-world
scenarios.