Bridging 3D Gaussian and Mesh for Freeview Video Rendering
Yuting Xiao Xuan Wang Jiafei Li Hongrui Cai Yanbo Fan Nan Xue Minghui Yang Yujun Shen Shenghua Gao
This is only a preview version of GauMesh. Recently, primitive-based
rendering has been proven to achieve convincing results in solving the problem
of modeling and rendering the 3D dynamic scene from 2D images. Despite this, in
the context of novel view synthesis, each type of primitive has its inherent
defects in terms of representation ability. It is difficult to exploit the mesh
to depict the fuzzy geometry. Meanwhile, the point-based splatting (e.g. the 3D
Gaussian Splatting) method usually produces artifacts or blurry pixels in the
area with smooth geometry and sharp textures. As a result, it is difficult,
even not impossible, to represent the complex and dynamic scene with a single
type of primitive. To this end, we propose a novel approach, GauMesh, to bridge
the 3D Gaussian and Mesh for modeling and rendering the dynamic scenes. Given a
sequence of tracked mesh as initialization, our goal is to simultaneously
optimize the mesh geometry, color texture, opacity maps, a set of 3D Gaussians,
and the deformation field. At a specific time, we perform $\alpha$-blending on
the RGB and opacity values based on the merged and re-ordered z-buffers from
mesh and 3D Gaussian rasterizations. This produces the final rendering, which
is supervised by the ground-truth image. Experiments demonstrate that our
approach adapts the appropriate type of primitives to represent the different
parts of the dynamic scene and outperforms all the baseline methods in both
quantitative and qualitative comparisons without losing render speed.