HFNeRF: Learning Human Biomechanic Features with Neural Radiance Fields
Arnab Dey Di Yang Antitza Dantcheva Jean Martinet
In recent advancements in novel view synthesis, generalizable Neural Radiance
Fields (NeRF) based methods applied to human subjects have shown remarkable
results in generating novel views from few images. However, this generalization
ability cannot capture the underlying structural features of the skeleton
shared across all instances. Building upon this, we introduce HFNeRF: a novel
generalizable human feature NeRF aimed at generating human biomechanic features
using a pre-trained image encoder. While previous human NeRF methods have shown
promising results in the generation of photorealistic virtual avatars, such
methods lack underlying human structure or biomechanic features such as
skeleton or joint information that are crucial for downstream applications
including Augmented Reality (AR)/Virtual Reality (VR). HFNeRF leverages 2D
pre-trained foundation models toward learning human features in 3D using neural
rendering, and then volume rendering towards generating 2D feature maps. We
evaluate HFNeRF in the skeleton estimation task by predicting heatmaps as
features. The proposed method is fully differentiable, allowing to successfully
learn color, geometry, and human skeleton in a simultaneous manner. This paper
presents preliminary results of HFNeRF, illustrating its potential in
generating realistic virtual avatars with biomechanic features using NeRF.