SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness
  for Hyperspectral Object Tracking
Hanzheng Wang Wei Li Xiang-Gen Xia Qian Du Jing Tian
Hyperspectral video (HSV) offers valuable spatial, spectral, and temporal
information simultaneously, making it highly suitable for handling challenges
such as background clutter and visual similarity in object tracking. However,
existing methods primarily focus on band regrouping and rely on RGB trackers
for feature extraction, resulting in limited exploration of spectral
information and difficulties in achieving complementary representations of
object features. In this paper, a spatial-spectral fusion network with spectral
angle awareness (SST-Net) is proposed for hyperspectral (HS) object tracking.
Firstly, to address the issue of insufficient spectral feature extraction in
existing networks, a spatial-spectral feature backbone ($S^2$FB) is designed.
With the spatial and spectral extraction branch, a joint representation of
texture and spectrum is obtained. Secondly, a spectral attention fusion module
(SAFM) is presented to capture the intra- and inter-modality correlation to
obtain the fused features from the HS and RGB modalities. It can incorporate
the visual information into the HS spectral context to form a robust
representation. Thirdly, to ensure a more accurate response of the tracker to
the object position, a spectral angle awareness module (SAAM) investigates the
region-level spectral similarity between the template and search images during
the prediction stage. Furthermore, we develop a novel spectral angle awareness
loss (SAAL) to offer guidance for the SAAM based on similar regions. Finally,
to obtain the robust tracking results, a weighted prediction method is
considered to combine the HS and RGB predicted motions of objects to leverage
the strengths of each modality. Extensive experiments on the HOTC dataset
demonstrate the effectiveness of the proposed SSF-Net, compared with
state-of-the-art trackers.