Joint 3D Shape and Motion Estimation from Rolling Shutter Light-Field
  Images
Hermes McGriff Renato Martins Nicolas Andreff Cédric Demonceaux
In this paper, we propose an approach to address the problem of 3D
reconstruction of scenes from a single image captured by a light-field camera
equipped with a rolling shutter sensor. Our method leverages the 3D information
cues present in the light-field and the motion information provided by the
rolling shutter effect. We present a generic model for the imaging process of
this sensor and a two-stage algorithm that minimizes the re-projection error
while considering the position and motion of the camera in a motion-shape
bundle adjustment estimation strategy. Thereby, we provide an instantaneous 3D
shape-and-pose-and-velocity sensing paradigm. To the best of our knowledge,
this is the first study to leverage this type of sensor for this purpose. We
also present a new benchmark dataset composed of different light-fields showing
rolling shutter effects, which can be used as a common base to improve the
evaluation and tracking the progress in the field. We demonstrate the
effectiveness and advantages of our approach through several experiments
conducted for different scenes and types of motions. The source code and
dataset are publicly available at: https://github.com/ICB-Vision-AI/RSLF