Bridging Sensor Gaps via Single-Direction Tuning for Hyperspectral Image
  Classification
Xizhe Xue Haokui Zhang Ying Li Liuwei Wan Zongwen Bai Mike Zheng Shou
Recently, some researchers started exploring the use of ViTs in tackling HSI
classification and achieved remarkable results. However, the training of ViT
models requires a considerable number of training samples, while hyperspectral
data, due to its high annotation costs, typically has a relatively small number
of training samples. This contradiction has not been effectively addressed. In
this paper, aiming to solve this problem, we propose the single-direction
tuning (SDT) strategy, which serves as a bridge, allowing us to leverage
existing labeled HSI datasets even RGB datasets to enhance the performance on
new HSI datasets with limited samples. The proposed SDT inherits the idea of
prompt tuning, aiming to reuse pre-trained models with minimal modifications
for adaptation to new tasks. But unlike prompt tuning, SDT is custom-designed
to accommodate the characteristics of HSIs. The proposed SDT utilizes a
parallel architecture, an asynchronous cold-hot gradient update strategy, and
unidirectional interaction. It aims to fully harness the potent representation
learning capabilities derived from training on heterologous, even cross-modal
datasets. In addition, we also introduce a novel Triplet-structured transformer
(Tri-Former), where spectral attention and spatial attention modules are merged
in parallel to construct the token mixing component for reducing computation
cost and a 3D convolution-based channel mixer module is integrated to enhance
stability and keep structure information. Comparison experiments conducted on
three representative HSI datasets captured by different sensors demonstrate the
proposed Tri-Former achieves better performance compared to several
state-of-the-art methods. Homologous, heterologous and cross-modal tuning
experiments verified the effectiveness of the proposed SDT.