Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation
Tairan He Zhengyi Luo Wenli Xiao Chong Zhang Kris Kitani Changliu Liu Guanya Shi
We present Human to Humanoid (H2O), a reinforcement learning (RL) based
framework that enables real-time whole-body teleoperation of a full-sized
humanoid robot with only an RGB camera. To create a large-scale retargeted
motion dataset of human movements for humanoid robots, we propose a scalable
"sim-to-data" process to filter and pick feasible motions using a privileged
motion imitator. Afterwards, we train a robust real-time humanoid motion
imitator in simulation using these refined motions and transfer it to the real
humanoid robot in a zero-shot manner. We successfully achieve teleoperation of
dynamic whole-body motions in real-world scenarios, including walking, back
jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our
knowledge, this is the first demonstration to achieve learning-based real-time
whole-body humanoid teleoperation.