Unfolding Framework with Prior of Convolution-Transformer Mixture and
  Uncertainty Estimation for Video Snapshot Compressive Imaging
Siming Zheng Xin Yuan
We consider the problem of video snapshot compressive imaging (SCI), where
sequential high-speed frames are modulated by different masks and captured by a
single measurement. The underlying principle of reconstructing multi-frame
images from only one single measurement is to solve an ill-posed problem. By
combining optimization algorithms and neural networks, deep unfolding networks
(DUNs) score tremendous achievements in solving inverse problems. In this
paper, our proposed model is under the DUN framework and we propose a 3D
Convolution-Transformer Mixture (CTM) module with a 3D efficient and scalable
attention model plugged in, which helps fully learn the correlation between
temporal and spatial dimensions by virtue of Transformer. To our best
knowledge, this is the first time that Transformer is employed to video SCI
reconstruction. Besides, to further investigate the high-frequency information
during the reconstruction process which are neglected in previous studies, we
introduce variance estimation characterizing the uncertainty on a
pixel-by-pixel basis. Extensive experimental results demonstrate that our
proposed method achieves state-of-the-art (SOTA) (with a 1.2dB gain in PSNR
over previous SOTA algorithm) results. We will release the code.