Recurrent Spike-based Image Restoration under General Illumination
Lin Zhu Yunlong Zheng Mengyue Geng Lizhi Wang Hua Huang
Spike camera is a new type of bio-inspired vision sensor that records light
intensity in the form of a spike array with high temporal resolution (20,000
Hz). This new paradigm of vision sensor offers significant advantages for many
vision tasks such as high speed image reconstruction. However, existing
spike-based approaches typically assume that the scenes are with sufficient
light intensity, which is usually unavailable in many real-world scenarios such
as rainy days or dusk scenes. To unlock more spike-based application scenarios,
we propose a Recurrent Spike-based Image Restoration (RSIR) network, which is
the first work towards restoring clear images from spike arrays under general
illumination. Specifically, to accurately describe the noise distribution under
different illuminations, we build a physical-based spike noise model according
to the sampling process of the spike camera. Based on the noise model, we
design our RSIR network which consists of an adaptive spike transformation
module, a recurrent temporal feature fusion module, and a frequency-based spike
denoising module. Our RSIR can process the spike array in a recursive manner to
ensure that the spike temporal information is well utilized. In the training
process, we generate the simulated spike data based on our noise model to train
our network. Extensive experiments on real-world datasets with different
illuminations demonstrate the effectiveness of the proposed network. The code
and dataset are released at https://github.com/BIT-Vision/RSIR.