Benchmarks and Challenges in Pose Estimation for Egocentric Hand
  Interactions with Objects
Zicong Fan Takehiko Ohkawa Linlin Yang Nie Lin Zhishan Zhou Shihao Zhou Jiajun Liang Zhong Gao Xuanyang Zhang Xue Zhang Fei Li Liu Zheng Feng Lu Karim Abou Zeid Bastian Leibe Jeongwan On Seungryul Baek Aditya Prakash Saurabh Gupta Kun He Yoichi Sato Otmar Hilliges Hyung Jin Chang Angela Yao
We interact with the world with our hands and see it through our own
(egocentric) perspective. A holistic 3D understanding of such interactions from
egocentric views is important for tasks in robotics, AR/VR, action recognition
and motion generation. Accurately reconstructing such interactions in 3D is
challenging due to heavy occlusion, viewpoint bias, camera distortion, and
motion blur from the head movement. To this end, we designed the HANDS23
challenge based on the AssemblyHands and ARCTIC datasets with carefully
designed training and testing splits. Based on the results of the top submitted
methods and more recent baselines on the leaderboards, we perform a thorough
analysis on 3D hand(-object) reconstruction tasks. Our analysis demonstrates
the effectiveness of addressing distortion specific to egocentric cameras,
adopting high-capacity transformers to learn complex hand-object interactions,
and fusing predictions from different views. Our study further reveals
challenging scenarios intractable with state-of-the-art methods, such as fast
hand motion, object reconstruction from narrow egocentric views, and close
contact between two hands and objects. Our efforts will enrich the community's
knowledge foundation and facilitate future hand studies on egocentric
hand-object interactions.