OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos
Dongyoung Choi Hyeonjoong Jang Min H. Kim
Omnidirectional cameras are extensively used in various applications to
provide a wide field of vision. However, they face a challenge in synthesizing
novel views due to the inevitable presence of dynamic objects, including the
photographer, in their wide field of view. In this paper, we introduce a new
approach called Omnidirectional Local Radiance Fields (OmniLocalRF) that can
render static-only scene views, removing and inpainting dynamic objects
simultaneously. Our approach combines the principles of local radiance fields
with the bidirectional optimization of omnidirectional rays. Our input is an
omnidirectional video, and we evaluate the mutual observations of the entire
angle between the previous and current frames. To reduce ghosting artifacts of
dynamic objects and inpaint occlusions, we devise a multi-resolution motion
mask prediction module. Unlike existing methods that primarily separate dynamic
components through the temporal domain, our method uses multi-resolution neural
feature planes for precise segmentation, which is more suitable for long
360-degree videos. Our experiments validate that OmniLocalRF outperforms
existing methods in both qualitative and quantitative metrics, especially in
scenarios with complex real-world scenes. In particular, our approach
eliminates the need for manual interaction, such as drawing motion masks by
hand and additional pose estimation, making it a highly effective and efficient
solution.