HandMIM: Pose-Aware Self-Supervised Learning for 3D Hand Mesh Estimation
Zuyan Liu Gaojie Lin Congyi Wang Min Zheng Feida Zhu
With an enormous number of hand images generated over time, unleashing pose
knowledge from unlabeled images for supervised hand mesh estimation is an
emerging yet challenging topic. To alleviate this issue, semi-supervised and
self-supervised approaches have been proposed, but they are limited by the
reliance on detection models or conventional ResNet backbones. In this paper,
inspired by the rapid progress of Masked Image Modeling (MIM) in visual
classification tasks, we propose a novel self-supervised pre-training strategy
for regressing 3D hand mesh parameters. Our approach involves a unified and
multi-granularity strategy that includes a pseudo keypoint alignment module in
the teacher-student framework for learning pose-aware semantic class tokens.
For patch tokens with detailed locality, we adopt a self-distillation manner
between teacher and student network based on MIM pre-training. To better fit
low-level regression tasks, we incorporate pixel reconstruction tasks for
multi-level representation learning. Additionally, we design a strong pose
estimation baseline using a simple vanilla vision Transformer (ViT) as the
backbone and attach a PyMAF head after tokens for regression. Extensive
experiments demonstrate that our proposed approach, named HandMIM, achieves
strong performance on various hand mesh estimation tasks. Notably, HandMIM
outperforms specially optimized architectures, achieving 6.29mm and 8.00mm
PAVPE (Vertex-Point-Error) on challenging FreiHAND and HO3Dv2 test sets,
respectively, establishing new state-of-the-art records on 3D hand mesh
estimation.