Video2Game: Real-time, Interactive, Realistic and Browser-Compatible
  Environment from a Single Video
Hongchi Xia Zhi-Hao Lin Wei-Chiu Ma Shenlong Wang
Creating high-quality and interactive virtual environments, such as games and
simulators, often involves complex and costly manual modeling processes. In
this paper, we present Video2Game, a novel approach that automatically converts
videos of real-world scenes into realistic and interactive game environments.
At the heart of our system are three core components:(i) a neural radiance
fields (NeRF) module that effectively captures the geometry and visual
appearance of the scene; (ii) a mesh module that distills the knowledge from
NeRF for faster rendering; and (iii) a physics module that models the
interactions and physical dynamics among the objects. By following the
carefully designed pipeline, one can construct an interactable and actionable
digital replica of the real world. We benchmark our system on both indoor and
large-scale outdoor scenes. We show that we can not only produce
highly-realistic renderings in real-time, but also build interactive games on
top.