PoseGen: Learning to Generate 3D Human Pose Dataset with NeRF
Mohsen Gholami Rabab Ward Z. Jane Wang
This paper proposes an end-to-end framework for generating 3D human pose
datasets using Neural Radiance Fields (NeRF). Public datasets generally have
limited diversity in terms of human poses and camera viewpoints, largely due to
the resource-intensive nature of collecting 3D human pose data. As a result,
pose estimators trained on public datasets significantly underperform when
applied to unseen out-of-distribution samples. Previous works proposed
augmenting public datasets by generating 2D-3D pose pairs or rendering a large
amount of random data. Such approaches either overlook image rendering or
result in suboptimal datasets for pre-trained models. Here we propose PoseGen,
which learns to generate a dataset (human 3D poses and images) with a feedback
loss from a given pre-trained pose estimator. In contrast to prior art, our
generated data is optimized to improve the robustness of the pre-trained model.
The objective of PoseGen is to learn a distribution of data that maximizes the
prediction error of a given pre-trained model. As the learned data distribution
contains OOD samples of the pre-trained model, sampling data from such a
distribution for further fine-tuning a pre-trained model improves the
generalizability of the model. This is the first work that proposes NeRFs for
3D human data generation. NeRFs are data-driven and do not require 3D scans of
humans. Therefore, using NeRF for data generation is a new direction for
convenient user-specific data generation. Our extensive experiments show that
the proposed PoseGen improves two baseline models (SPIN and HybrIK) on four
datasets with an average 6% relative improvement.