TimeRewind: Rewinding Time with Image-and-Events Video Diffusion
Jingxi Chen Brandon Y. Feng Haoming Cai Mingyang Xie Christopher Metzler Cornelia Fermuller Yiannis Aloimonos
This paper addresses the novel challenge of ``rewinding'' time from a single
captured image to recover the fleeting moments missed just before the shutter
button is pressed. This problem poses a significant challenge in computer
vision and computational photography, as it requires predicting plausible
pre-capture motion from a single static frame, an inherently ill-posed task due
to the high degree of freedom in potential pixel movements. We overcome this
challenge by leveraging the emerging technology of neuromorphic event cameras,
which capture motion information with high temporal resolution, and integrating
this data with advanced image-to-video diffusion models. Our proposed framework
introduces an event motion adaptor conditioned on event camera data, guiding
the diffusion model to generate videos that are visually coherent and
physically grounded in the captured events. Through extensive experimentation,
we demonstrate the capability of our approach to synthesize high-quality videos
that effectively ``rewind'' time, showcasing the potential of combining event
camera technology with generative models. Our work opens new avenues for
research at the intersection of computer vision, computational photography, and
generative modeling, offering a forward-thinking solution to capturing missed
moments and enhancing future consumer cameras and smartphones. Please see the
project page at https://timerewind.github.io/ for video results and code
release.