Scene-Generalizable Interactive Segmentation of Radiance Fields
Songlin Tang Wenjie Pei Xin Tao Tanghui Jia Guangming Lu Yu-Wing Tai
Existing methods for interactive segmentation in radiance fields entail
scene-specific optimization and thus cannot generalize across different scenes,
which greatly limits their applicability. In this work we make the first
attempt at Scene-Generalizable Interactive Segmentation in Radiance Fields
(SGISRF) and propose a novel SGISRF method, which can perform 3D object
segmentation for novel (unseen) scenes represented by radiance fields, guided
by only a few interactive user clicks in a given set of multi-view 2D images.
In particular, the proposed SGISRF focuses on addressing three crucial
challenges with three specially designed techniques. First, we devise the
Cross-Dimension Guidance Propagation to encode the scarce 2D user clicks into
informative 3D guidance representations. Second, the Uncertainty-Eliminated 3D
Segmentation module is designed to achieve efficient yet effective 3D
segmentation. Third, Concealment-Revealed Supervised Learning scheme is
proposed to reveal and correct the concealed 3D segmentation errors resulted
from the supervision in 2D space with only 2D mask annotations. Extensive
experiments on two real-world challenging benchmarks covering diverse scenes
demonstrate 1) effectiveness and scene-generalizability of the proposed method,
2) favorable performance compared to classical method requiring scene-specific
optimization.