Can Transformers Capture Spatial Relations between Objects?
Chuan Wen Dinesh Jayaraman Yang Gao
Spatial relationships between objects represent key scene information for
humans to understand and interact with the world. To study the capability of
current computer vision systems to recognize physically grounded spatial
relations, we start by proposing precise relation definitions that permit
consistently annotating a benchmark dataset. Despite the apparent simplicity of
this task relative to others in the recognition literature, we observe that
existing approaches perform poorly on this benchmark. We propose new approaches
exploiting the long-range attention capabilities of transformers for this task,
and evaluating key design principles. We identify a simple "RelatiViT"
architecture and demonstrate that it outperforms all current approaches. To our
knowledge, this is the first method to convincingly outperform naive baselines
on spatial relation prediction in in-the-wild settings. The code and datasets
are available in \url{https://sites.google.com/view/spatial-relation}.