Virtual Pets: Animatable Animal Generation in 3D Scenes
Yen-Chi Cheng Chieh Hubert Lin Chaoyang Wang Yash Kant Sergey Tulyakov Alexander Schwing Liangyan Gui Hsin-Ying Lee
Toward unlocking the potential of generative models in immersive 4D
experiences, we introduce Virtual Pet, a novel pipeline to model realistic and
diverse motions for target animal species within a 3D environment. To
circumvent the limited availability of 3D motion data aligned with
environmental geometry, we leverage monocular internet videos and extract
deformable NeRF representations for the foreground and static NeRF
representations for the background. For this, we develop a reconstruction
strategy, encompassing species-level shared template learning and per-video
fine-tuning. Utilizing the reconstructed data, we then train a conditional 3D
motion model to learn the trajectory and articulation of foreground animals in
the context of 3D backgrounds. We showcase the efficacy of our pipeline with
comprehensive qualitative and quantitative evaluations using cat videos. We
also demonstrate versatility across unseen cats and indoor environments,
producing temporally coherent 4D outputs for enriched virtual experiences.