Seeing Beyond the Patch: Scale-Adaptive Semantic Segmentation of
  High-resolution Remote Sensing Imagery based on Reinforcement Learning
Yinhe Liu Sunan Shi Junjue Wang Yanfei Zhong
In remote sensing imagery analysis, patch-based methods have limitations in
capturing information beyond the sliding window. This shortcoming poses a
significant challenge in processing complex and variable geo-objects, which
results in semantic inconsistency in segmentation results. To address this
challenge, we propose a dynamic scale perception framework, named GeoAgent,
which adaptively captures appropriate scale context information outside the
image patch based on the different geo-objects. In GeoAgent, each image patch's
states are represented by a global thumbnail and a location mask. The global
thumbnail provides context beyond the patch, and the location mask guides the
perceived spatial relationships. The scale-selection actions are performed
through a Scale Control Agent (SCA). A feature indexing module is proposed to
enhance the ability of the agent to distinguish the current image patch's
location. The action switches the patch scale and context branch of a
dual-branch segmentation network that extracts and fuses the features of
multi-scale patches. The GeoAgent adjusts the network parameters to perform the
appropriate scale-selection action based on the reward received for the
selected scale. The experimental results, using two publicly available datasets
and our newly constructed dataset WUSU, demonstrate that GeoAgent outperforms
previous segmentation methods, particularly for large-scale mapping
applications.