General Flow as Foundation Affordance for Scalable Robot Learning
Chengbo Yuan Chuan Wen Tong Zhang Yang Gao
We address the challenge of acquiring real-world manipulation skills with a
scalable framework.Inspired by the success of large-scale auto-regressive
prediction in Large Language Models (LLMs), we hold the belief that identifying
an appropriate prediction target capable of leveraging large-scale datasets is
crucial for achieving efficient and universal learning. Therefore, we propose
to utilize flow, which represents the future trajectories of 3D points on
objects of interest, as an ideal prediction target in robot learning. To
exploit scalable data resources, we turn our attention to cross-embodiment
datasets. We develop, for the first time, a language-conditioned prediction
model directly from large-scale RGBD human video datasets. Our predicted flow
offers actionable geometric and physics guidance, thus facilitating stable
zero-shot skill transfer in real-world scenarios.We deploy our method with a
policy based on closed-loop flow prediction. Remarkably, without any additional
training, our method achieves an impressive 81% success rate in human-to-robot
skill transfer, covering 18 tasks in 6 scenes. Our framework features the
following benefits: (1) scalability: leveraging cross-embodiment data
resources; (2) universality: multiple object categories, including rigid,
articulated, and soft bodies; (3) stable skill transfer: providing actionable
guidance with a small inference domain-gap. These lead to a new pathway towards
scalable general robot learning. Data, code, and model weights will be made
publicly available.