UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures
  for Human Avatar Modeling
Yujiao Jiang Qingmin Liao Xiaoyu Li Li Ma Qi Zhang Chaopeng Zhang Zongqing Lu Ying Shan
Reconstructing photo-realistic drivable human avatars from multi-view image
sequences has been a popular and challenging topic in the field of computer
vision and graphics. While existing NeRF-based methods can achieve high-quality
novel view rendering of human models, both training and inference processes are
time-consuming. Recent approaches have utilized 3D Gaussians to represent the
human body, enabling faster training and rendering. However, they undermine the
importance of the mesh guidance and directly predict Gaussians in 3D space with
coarse mesh guidance. This hinders the learning procedure of the Gaussians and
tends to produce blurry textures. Therefore, we propose UV Gaussians, which
models the 3D human body by jointly learning mesh deformations and 2D UV-space
Gaussian textures. We utilize the embedding of UV map to learn Gaussian
textures in 2D space, leveraging the capabilities of powerful 2D networks to
extract features. Additionally, through an independent Mesh network, we
optimize pose-dependent geometric deformations, thereby guiding Gaussian
rendering and significantly enhancing rendering quality. We collect and process
a new dataset of human motion, which includes multi-view images, scanned
models, parametric model registration, and corresponding texture maps.
Experimental results demonstrate that our method achieves state-of-the-art
synthesis of novel view and novel pose. The code and data will be made
available on the homepage https://alex-jyj.github.io/UV-Gaussians/ once the
paper is accepted.