SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image
Yunhao Li Xiaodong Wang Ping Wang Xin Yuan Peidong Liu
In this paper, we explore the potential of Snapshot Compressive Imaging (SCI)
technique for recovering the underlying 3D scene representation from a single
temporal compressed image. SCI is a cost-effective method that enables the
recording of high-dimensional data, such as hyperspectral or temporal
information, into a single image using low-cost 2D imaging sensors. To achieve
this, a series of specially designed 2D masks are usually employed, which not
only reduces storage requirements but also offers potential privacy protection.
Inspired by this, to take one step further, our approach builds upon the
powerful 3D scene representation capabilities of neural radiance fields (NeRF).
Specifically, we formulate the physical imaging process of SCI as part of the
training of NeRF, allowing us to exploit its impressive performance in
capturing complex scene structures. To assess the effectiveness of our method,
we conduct extensive evaluations using both synthetic data and real data
captured by our SCI system. Extensive experimental results demonstrate that our
proposed approach surpasses the state-of-the-art methods in terms of image
reconstruction and novel view image synthesis. Moreover, our method also
exhibits the ability to restore high frame-rate multi-view consistent images by
leveraging SCI and the rendering capabilities of NeRF. The code is available at
https://github.com/WU-CVGL/SCINeRF.