Panoptic Perception: A Novel Task and Fine-grained Dataset for Universal
  Remote Sensing Image Interpretation
Danpei Zhao Bo Yuan Ziqiang Chen Tian Li Zhuoran Liu Wentao Li Yue Gao
Current remote-sensing interpretation models often focus on a single task
such as detection, segmentation, or caption. However, the task-specific
designed models are unattainable to achieve the comprehensive multi-level
interpretation of images. The field also lacks support for multi-task joint
interpretation datasets. In this paper, we propose Panoptic Perception, a novel
task and a new fine-grained dataset (FineGrip) to achieve a more thorough and
universal interpretation for RSIs. The new task, 1) integrates pixel-level,
instance-level, and image-level information for universal image perception, 2)
captures image information from coarse to fine granularity, achieving deeper
scene understanding and description, and 3) enables various independent tasks
to complement and enhance each other through multi-task learning. By
emphasizing multi-task interactions and the consistency of perception results,
this task enables the simultaneous processing of fine-grained foreground
instance segmentation, background semantic segmentation, and global
fine-grained image captioning. Concretely, the FineGrip dataset includes 2,649
remote sensing images, 12,054 fine-grained instance segmentation masks
belonging to 20 foreground things categories, 7,599 background semantic masks
for 5 stuff classes and 13,245 captioning sentences. Furthermore, we propose a
joint optimization-based panoptic perception model. Experimental results on
FineGrip demonstrate the feasibility of the panoptic perception task and the
beneficial effect of multi-task joint optimization on individual tasks. The
dataset will be publicly available.