Reconstructing Interacting Hands with Interaction Prior from Monocular
  Images
Binghui Zuo Zimeng Zhao Wenqian Sun Wei Xie Zhou Xue Yangang Wang
Reconstructing interacting hands from monocular images is indispensable in
AR/VR applications. Most existing solutions rely on the accurate localization
of each skeleton joint. However, these methods tend to be unreliable due to the
severe occlusion and confusing similarity among adjacent hand parts. This also
defies human perception because humans can quickly imitate an interaction
pattern without localizing all joints. Our key idea is to first construct a
two-hand interaction prior and recast the interaction reconstruction task as
the conditional sampling from the prior. To expand more interaction states, a
large-scale multimodal dataset with physical plausibility is proposed. Then a
VAE is trained to further condense these interaction patterns as latent codes
in a prior distribution. When looking for image cues that contribute to
interaction prior sampling, we propose the interaction adjacency heatmap (IAH).
Compared with a joint-wise heatmap for localization, IAH assigns denser visible
features to those invisible joints. Compared with an all-in-one visible
heatmap, it provides more fine-grained local interaction information in each
interaction region. Finally, the correlations between the extracted features
and corresponding interaction codes are linked by the ViT module. Comprehensive
evaluations on benchmark datasets have verified the effectiveness of this
framework. The code and dataset are publicly available at
https://github.com/binghui-z/InterPrior_pytorch