Extract-and-Adaptation Network for 3D Interacting Hand Mesh Recovery
JoonKyu Park Daniel Sungho Jung Gyeongsik Moon Kyoung Mu Lee
Understanding how two hands interact with each other is a key component of
accurate 3D interacting hand mesh recovery. However, recent Transformer-based
methods struggle to learn the interaction between two hands as they directly
utilize two hand features as input tokens, which results in distant token
problem. The distant token problem represents that input tokens are in
heterogeneous spaces, leading Transformer to fail in capturing correlation
between input tokens. Previous Transformer-based methods suffer from the
problem especially when poses of two hands are very different as they project
features from a backbone to separate left and right hand-dedicated features. We
present EANet, extract-and-adaptation network, with EABlock, the main component
of our network. Rather than directly utilizing two hand features as input
tokens, our EABlock utilizes two complementary types of novel tokens, SimToken
and JoinToken, as input tokens. Our two novel tokens are from a combination of
separated two hand features; hence, it is much more robust to the distant token
problem. Using the two type of tokens, our EABlock effectively extracts
interaction feature and adapts it to each hand. The proposed EANet achieves the
state-of-the-art performance on 3D interacting hands benchmarks. The codes are
available at https://github.com/jkpark0825/EANet.