Inherent Redundancy in Spiking Neural Networks
Man Yao Jiakui Hu Guangshe Zhao Yaoyuan Wang Ziyang Zhang Bo Xu Guoqi Li
Spiking Neural Networks (SNNs) are well known as a promising energy-efficient
alternative to conventional artificial neural networks. Subject to the
preconceived impression that SNNs are sparse firing, the analysis and
optimization of inherent redundancy in SNNs have been largely overlooked, thus
the potential advantages of spike-based neuromorphic computing in accuracy and
energy efficiency are interfered. In this work, we pose and focus on three key
questions regarding the inherent redundancy in SNNs. We argue that the
redundancy is induced by the spatio-temporal invariance of SNNs, which enhances
the efficiency of parameter utilization but also invites lots of noise spikes.
Further, we analyze the effect of spatio-temporal invariance on the
spatio-temporal dynamics and spike firing of SNNs. Then, motivated by these
analyses, we propose an Advance Spatial Attention (ASA) module to harness SNNs'
redundancy, which can adaptively optimize their membrane potential distribution
by a pair of individual spatial attention sub-modules. In this way, noise spike
features are accurately regulated. Experimental results demonstrate that the
proposed method can significantly drop the spike firing with better performance
than state-of-the-art SNN baselines. Our code is available in
\url{https://github.com/BICLab/ASA-SNN}.