Mesh-based Gaussian Splatting for Real-time Large-scale Deformation
Lin Gao Jie Yang Bo-Tao Zhang Jia-Mu Sun Yu-Jie Yuan Hongbo Fu Yu-Kun Lai
Neural implicit representations, including Neural Distance Fields and Neural
Radiance Fields, have demonstrated significant capabilities for reconstructing
surfaces with complicated geometry and topology, and generating novel views of
a scene. Nevertheless, it is challenging for users to directly deform or
manipulate these implicit representations with large deformations in the
real-time fashion. Gaussian Splatting(GS) has recently become a promising
method with explicit geometry for representing static scenes and facilitating
high-quality and real-time synthesis of novel views. However,it cannot be
easily deformed due to the use of discrete Gaussians and lack of explicit
topology. To address this, we develop a novel GS-based method that enables
interactive deformation. Our key idea is to design an innovative mesh-based GS
representation, which is integrated into Gaussian learning and manipulation. 3D
Gaussians are defined over an explicit mesh, and they are bound with each
other: the rendering of 3D Gaussians guides the mesh face split for adaptive
refinement, and the mesh face split directs the splitting of 3D Gaussians.
Moreover, the explicit mesh constraints help regularize the Gaussian
distribution, suppressing poor-quality Gaussians(e.g. misaligned
Gaussians,long-narrow shaped Gaussians), thus enhancing visual quality and
avoiding artifacts during deformation. Based on this representation, we further
introduce a large-scale Gaussian deformation technique to enable deformable GS,
which alters the parameters of 3D Gaussians according to the manipulation of
the associated mesh. Our method benefits from existing mesh deformation
datasets for more realistic data-driven Gaussian deformation. Extensive
experiments show that our approach achieves high-quality reconstruction and
effective deformation, while maintaining the promising rendering results at a
high frame rate(65 FPS on average).