Structured World Models from Human Videos
Russell Mendonca Shikhar Bahl Deepak Pathak
We tackle the problem of learning complex, general behaviors directly in the
real world. We propose an approach for robots to efficiently learn manipulation
skills using only a handful of real-world interaction trajectories from many
different settings. Inspired by the success of learning from large-scale
datasets in the fields of computer vision and natural language, our belief is
that in order to efficiently learn, a robot must be able to leverage
internet-scale, human video data. Humans interact with the world in many
interesting ways, which can allow a robot to not only build an understanding of
useful actions and affordances but also how these actions affect the world for
manipulation. Our approach builds a structured, human-centric action space
grounded in visual affordances learned from human videos. Further, we train a
world model on human videos and fine-tune on a small amount of robot
interaction data without any task supervision. We show that this approach of
affordance-space world models enables different robots to learn various
manipulation skills in complex settings, in under 30 minutes of interaction.
Videos can be found at https://human-world-model.github.io