RH20T-P: A Primitive-Level Robotic Dataset Towards Composable
  Generalization Agents
Zeren Chen Zhelun Shi Xiaoya Lu Lehan He Sucheng Qian Hao Shu Fang Zhenfei Yin Wanli Ouyang Jing Shao Yu Qiao Cewu Lu Lu Sheng
The ultimate goals of robotic learning is to acquire a comprehensive and
generalizable robotic system capable of performing both seen skills within the
training distribution and unseen skills in novel environments. Recent progress
in utilizing language models as high-level planners has demonstrated that the
complexity of tasks can be reduced through decomposing them into
primitive-level plans, making it possible to generalize on novel robotic tasks
in a composable manner. Despite the promising future, the community is not yet
adequately prepared for composable generalization agents, particularly due to
the lack of primitive-level real-world robotic datasets. In this paper, we
propose a primitive-level robotic dataset, namely RH20T-P, which contains about
33000 video clips covering 44 diverse and complicated robotic tasks. Each clip
is manually annotated according to a set of meticulously designed primitive
skills, facilitating the future development of composable generalization
agents. To validate the effectiveness of RH20T-P, we also construct a potential
and scalable agent based on RH20T-P, called RA-P. Equipped with two planners
specialized in task decomposition and motion planning, RA-P can adapt to novel
physical skills through composable generalization. Our website and videos can
be found at https://sites.google.com/view/rh20t-primitive/main. Dataset and
code will be made available soon.