Finding Visual Saliency in Continuous Spike Stream
Lin Zhu Xianzhang Chen Xiao Wang Hua Huang
As a bio-inspired vision sensor, the spike camera emulates the operational
principles of the fovea, a compact retinal region, by employing spike
discharges to encode the accumulation of per-pixel luminance intensity.
Leveraging its high temporal resolution and bio-inspired neuromorphic design,
the spike camera holds significant promise for advancing computer vision
applications. Saliency detection mimics the behavior of human beings and
captures the most salient region from the scenes. In this paper, we investigate
the visual saliency in the continuous spike stream for the first time. To
effectively process the binary spike stream, we propose a Recurrent Spiking
Transformer (RST) framework, which is based on a full spiking neural network.
Our framework enables the extraction of spatio-temporal features from the
continuous spatio-temporal spike stream while maintaining low power
consumption. To facilitate the training and validation of our proposed model,
we build a comprehensive real-world spike-based visual saliency dataset,
enriched with numerous light conditions. Extensive experiments demonstrate the
superior performance of our Recurrent Spiking Transformer framework in
comparison to other spike neural network-based methods. Our framework exhibits
a substantial margin of improvement in capturing and highlighting visual
saliency in the spike stream, which not only provides a new perspective for
spike-based saliency segmentation but also shows a new paradigm for full
SNN-based transformer models. The code and dataset are available at
\url{https://github.com/BIT-Vision/SVS}.