Replay: Multi-modal Multi-view Acted Videos for Casual Holography
Roman Shapovalov Yanir Kleiman Ignacio Rocco David Novotny Andrea Vedaldi Changan Chen Filippos Kokkinos Ben Graham Natalia Neverova
We introduce Replay, a collection of multi-view, multi-modal videos of humans
interacting socially. Each scene is filmed in high production quality, from
different viewpoints with several static cameras, as well as wearable action
cameras, and recorded with a large array of microphones at different positions
in the room. Overall, the dataset contains over 4000 minutes of footage and
over 7 million timestamped high-resolution frames annotated with camera poses
and partially with foreground masks. The Replay dataset has many potential
applications, such as novel-view synthesis, 3D reconstruction, novel-view
acoustic synthesis, human body and face analysis, and training generative
models. We provide a benchmark for training and evaluating novel-view
synthesis, with two scenarios of different difficulty. Finally, we evaluate
several baseline state-of-the-art methods on the new benchmark.