Learning transformer-based heterogeneously salient graph representation
  for multimodal fusion classification of hyperspectral image and LiDAR data
Jiaqi Yang Bo Du Liangpei Zhang
Data collected by different modalities can provide a wealth of complementary
information, such as hyperspectral image (HSI) to offer rich spectral-spatial
properties, synthetic aperture radar (SAR) to provide structural information
about the Earth's surface, and light detection and ranging (LiDAR) to cover
altitude information about ground elevation. Therefore, a natural idea is to
combine multimodal images for refined and accurate land-cover interpretation.
Although many efforts have been attempted to achieve multi-source remote
sensing image classification, there are still three issues as follows: 1)
indiscriminate feature representation without sufficiently considering modal
heterogeneity, 2) abundant features and complex computations associated with
modeling long-range dependencies, and 3) overfitting phenomenon caused by
sparsely labeled samples. To overcome the above barriers, a transformer-based
heterogeneously salient graph representation (THSGR) approach is proposed in
this paper. First, a multimodal heterogeneous graph encoder is presented to
encode distinctively non-Euclidean structural features from heterogeneous data.
Then, a self-attention-free multi-convolutional modulator is designed for
effective and efficient long-term dependency modeling. Finally, a mean forward
is put forward in order to avoid overfitting. Based on the above structures,
the proposed model is able to break through modal gaps to obtain differentiated
graph representation with competitive time cost, even for a small fraction of
training samples. Experiments and analyses on three benchmark datasets with
various state-of-the-art (SOTA) methods show the performance of the proposed
approach.