Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos
Matthew Chang Aditya Prakash Saurabh Gupta
The analysis and use of egocentric videos for robotic tasks is made
challenging by occlusion due to the hand and the visual mismatch between the
human hand and a robot end-effector. In this sense, the human hand presents a
nuisance. However, often hands also provide a valuable signal, e.g. the hand
pose may suggest what kind of object is being held. In this work, we propose to
extract a factored representation of the scene that separates the agent (human
hand) and the environment. This alleviates both occlusion and mismatch while
preserving the signal, thereby easing the design of models for downstream
robotics tasks. At the heart of this factorization is our proposed Video
Inpainting via Diffusion Model (VIDM) that leverages both a prior on real-world
images (through a large-scale pre-trained diffusion model) and the appearance
of the object in earlier frames of the video (through attention). Our
experiments demonstrate the effectiveness of VIDM at improving inpainting
quality on egocentric videos and the power of our factored representation for
numerous tasks: object detection, 3D reconstruction of manipulated objects, and
learning of reward functions, policies, and affordances from videos.