EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote
  Sensing Visual Question Answering
Junjue Wang Zhuo Zheng Zihang Chen Ailong Ma Yanfei Zhong
Earth vision research typically focuses on extracting geospatial object
locations and categories but neglects the exploration of relations between
objects and comprehensive reasoning. Based on city planning needs, we develop a
multi-modal multi-task VQA dataset (EarthVQA) to advance relational
reasoning-based judging, counting, and comprehensive analysis. The EarthVQA
dataset contains 6000 images, corresponding semantic masks, and 208,593 QA
pairs with urban and rural governance requirements embedded. As objects are the
basis for complex relational reasoning, we propose a Semantic OBject Awareness
framework (SOBA) to advance VQA in an object-centric way. To preserve refined
spatial locations and semantics, SOBA leverages a segmentation network for
object semantics generation. The object-guided attention aggregates object
interior features via pseudo masks, and bidirectional cross-attention further
models object external relations hierarchically. To optimize object counting,
we propose a numerical difference loss that dynamically adds difference
penalties, unifying the classification and regression tasks. Experimental
results show that SOBA outperforms both advanced general and remote sensing
methods. We believe this dataset and framework provide a strong benchmark for
Earth vision's complex analysis. The project page is at
https://Junjue-Wang.github.io/homepage/EarthVQA.