Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot
Fabien Baradel Matthieu Armando Salma Galaaoui Romain Brégier Philippe Weinzaepfel Grégory Rogez Thomas Lucas
We present Multi-HMR, a strong single-shot model for multi-person 3D human
mesh recovery from a single RGB image. Predictions encompass the whole body,
i.e, including hands and facial expressions, using the SMPL-X parametric model
and spatial location in the camera coordinate system. Our model detects people
by predicting coarse 2D heatmaps of person centers, using features produced by
a standard Vision Transformer (ViT) backbone. It then predicts their whole-body
pose, shape and spatial location using a new cross-attention module called the
Human Prediction Head (HPH), with one query per detected center token,
attending to the entire set of features. As direct prediction of SMPL-X
parameters yields suboptimal results, we introduce CUFFS; the Close-Up Frames
of Full-Body Subjects dataset, containing humans close to the camera with
diverse hand poses. We show that incorporating this dataset into training
further enhances predictions, particularly for hands, enabling us to achieve
state-of-the-art performance. Multi-HMR also optionally accounts for camera
intrinsics, if available, by encoding camera ray directions for each image
token. This simple design achieves strong performance on whole-body and
body-only benchmarks simultaneously. We train models with various backbone
sizes and input resolutions. In particular, using a ViT-S backbone and
$448\times448$ input images already yields a fast and competitive model with
respect to state-of-the-art methods, while considering larger models and higher
resolutions further improve performance.