RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for
  Robotic Manipulation
Hanxiao Jiang Binghao Huang Ruihai Wu Zhuoran Li Shubham Garg Hooshang Nayyeri Shenlong Wang Yunzhu Li
Robots need to explore their surroundings to adapt to and tackle tasks in
unknown environments. Prior work has proposed building scene graphs of the
environment but typically assumes that the environment is static, omitting
regions that require active interactions. This severely limits their ability to
handle more complex tasks in household and office environments: before setting
up a table, robots must explore drawers and cabinets to locate all utensils and
condiments. In this work, we introduce the novel task of interactive scene
exploration, wherein robots autonomously explore environments and produce an
action-conditioned scene graph (ACSG) that captures the structure of the
underlying environment. The ACSG accounts for both low-level information, such
as geometry and semantics, and high-level information, such as the
action-conditioned relationships between different entities in the scene. To
this end, we present the Robotic Exploration (RoboEXP) system, which
incorporates the Large Multimodal Model (LMM) and an explicit memory design to
enhance our system's capabilities. The robot reasons about what and how to
explore an object, accumulating new information through the interaction process
and incrementally constructing the ACSG. We apply our system across various
real-world settings in a zero-shot manner, demonstrating its effectiveness in
exploring and modeling environments it has never seen before. Leveraging the
constructed ACSG, we illustrate the effectiveness and efficiency of our RoboEXP
system in facilitating a wide range of real-world manipulation tasks involving
rigid, articulated objects, nested objects like Matryoshka dolls, and
deformable objects like cloth.