A Spatial-Temporal Dual-Mode Mixed Flow Network for Panoramic Video
  Salient Object Detection
Xiaolei Chen Pengcheng Zhang Zelong Du Ishfaq Ahmad
Salient object detection (SOD) in panoramic video is still in the initial
exploration stage. The indirect application of 2D video SOD method to the
detection of salient objects in panoramic video has many unmet challenges, such
as low detection accuracy, high model complexity, and poor generalization
performance. To overcome these hurdles, we design an Inter-Layer Attention
(ILA) module, an Inter-Layer weight (ILW) module, and a Bi-Modal Attention
(BMA) module. Based on these modules, we propose a Spatial-Temporal Dual-Mode
Mixed Flow Network (STDMMF-Net) that exploits the spatial flow of panoramic
video and the corresponding optical flow for SOD. First, the ILA module
calculates the attention between adjacent level features of consecutive frames
of panoramic video to improve the accuracy of extracting salient object
features from the spatial flow. Then, the ILW module quantifies the salient
object information contained in the features of each level to improve the
fusion efficiency of the features of each level in the mixed flow. Finally, the
BMA module improves the detection accuracy of STDMMF-Net. A large number of
subjective and objective experimental results testify that the proposed method
demonstrates better detection accuracy than the state-of-the-art (SOTA)
methods. Moreover, the comprehensive performance of the proposed method is
better in terms of memory required for model inference, testing time,
complexity, and generalization performance.