DexCap: Scalable and Portable Mocap Data Collection System for Dexterous
  Manipulation
Chen Wang Haochen Shi Weizhuo Wang Ruohan Zhang Li Fei-Fei C. Karen Liu
Imitation learning from human hand motion data presents a promising avenue
for imbuing robots with human-like dexterity in real-world manipulation tasks.
Despite this potential, substantial challenges persist, particularly with the
portability of existing hand motion capture (mocap) systems and the difficulty
of translating mocap data into effective control policies. To tackle these
issues, we introduce DexCap, a portable hand motion capture system, alongside
DexIL, a novel imitation algorithm for training dexterous robot skills directly
from human hand mocap data. DexCap offers precise, occlusion-resistant tracking
of wrist and finger motions based on SLAM and electromagnetic field together
with 3D observations of the environment. Utilizing this rich dataset, DexIL
employs inverse kinematics and point cloud-based imitation learning to
replicate human actions with robot hands. Beyond learning from human motion,
DexCap also offers an optional human-in-the-loop correction mechanism to refine
and further improve robot performance. Through extensive evaluation across six
dexterous manipulation tasks, our approach not only demonstrates superior
performance but also showcases the system's capability to effectively learn
from in-the-wild mocap data, paving the way for future data collection methods
for dexterous manipulation. More details can be found at
https://dex-cap.github.io