Score-Guided Diffusion for 3D Human Recovery
Anastasis Stathopoulos Ligong Han Dimitris Metaxas
We present Score-Guided Human Mesh Recovery (ScoreHMR), an approach for
solving inverse problems for 3D human pose and shape reconstruction. These
inverse problems involve fitting a human body model to image observations,
traditionally solved through optimization techniques. ScoreHMR mimics model
fitting approaches, but alignment with the image observation is achieved
through score guidance in the latent space of a diffusion model. The diffusion
model is trained to capture the conditional distribution of the human model
parameters given an input image. By guiding its denoising process with a
task-specific score, ScoreHMR effectively solves inverse problems for various
applications without the need for retraining the task-agnostic diffusion model.
We evaluate our approach on three settings/applications. These are: (i)
single-frame model fitting; (ii) reconstruction from multiple uncalibrated
views; (iii) reconstructing humans in video sequences. ScoreHMR consistently
outperforms all optimization baselines on popular benchmarks across all
settings. We make our code and models available at the
https://statho.github.io/ScoreHMR.