Panoramas from Photons
Sacha Jungerman Atul Ingle Mohit Gupta
Scene reconstruction in the presence of high-speed motion and low
illumination is important in many applications such as augmented and virtual
reality, drone navigation, and autonomous robotics. Traditional motion
estimation techniques fail in such conditions, suffering from too much blur in
the presence of high-speed motion and strong noise in low-light conditions.
Single-photon cameras have recently emerged as a promising technology capable
of capturing hundreds of thousands of photon frames per second thanks to their
high speed and extreme sensitivity. Unfortunately, traditional computer vision
techniques are not well suited for dealing with the binary-valued photon data
captured by these cameras because these are corrupted by extreme Poisson noise.
Here we present a method capable of estimating extreme scene motion under
challenging conditions, such as low light or high dynamic range, from a
sequence of high-speed image frames such as those captured by a single-photon
camera. Our method relies on iteratively improving a motion estimate by
grouping and aggregating frames after-the-fact, in a stratified manner. We
demonstrate the creation of high-quality panoramas under fast motion and
extremely low light, and super-resolution results using a custom single-photon
camera prototype. For code and supplemental material see our
$\href{https://wisionlab.com/project/panoramas-from-photons/}{\text{project
webpage}}$.