UniST: Towards Unifying Saliency Transformer for Video Saliency
  Prediction and Detection
Junwen Xiong Peng Zhang Chuanyue Li Wei Huang Yufei Zha Tao You
Video saliency prediction and detection are thriving research domains that
enable computers to simulate the distribution of visual attention akin to how
humans perceiving dynamic scenes. While many approaches have crafted
task-specific training paradigms for either video saliency prediction or video
salient object detection tasks, few attention has been devoted to devising a
generalized saliency modeling framework that seamlessly bridges both these
distinct tasks. In this study, we introduce the Unified Saliency Transformer
(UniST) framework, which comprehensively utilizes the essential attributes of
video saliency prediction and video salient object detection. In addition to
extracting representations of frame sequences, a saliency-aware transformer is
designed to learn the spatio-temporal representations at progressively
increased resolutions, while incorporating effective cross-scale saliency
information to produce a robust representation. Furthermore, a task-specific
decoder is proposed to perform the final prediction for each task. To the best
of our knowledge, this is the first work that explores designing a transformer
structure for both saliency modeling tasks. Convincible experiments demonstrate
that the proposed UniST achieves superior performance across seven challenging
benchmarks for two tasks, and significantly outperforms the other
state-of-the-art methods.