Grasp, See and Place: Efficient Unknown Object Rearrangement with Policy
  Structure Prior
Kechun Xu Zhongxiang Zhou Jun Wu Haojian Lu Rong Xiong Yue Wang
We focus on the task of unknown object rearrangement, where a robot is
supposed to re-configure the objects into a desired goal configuration
specified by an RGB-D image. Recent works explore unknown object rearrangement
systems by incorporating learning-based perception modules. However, they are
sensitive to perception error, and pay less attention to task-level
performance. In this paper, we aim to develop an effective system for unknown
object rearrangement amidst perception noise. We theoretically reveal the noisy
perception impacts grasp and place in a decoupled way, and show such a
decoupled structure is non-trivial to improve task optimality. We propose GSP,
a dual-loop system with the decoupled structure as prior. For the inner loop,
we learn an active seeing policy for self-confident object matching to improve
the perception of place. For the outer loop, we learn a grasp policy aware of
object matching and grasp capability guided by task-level rewards. We leverage
the foundation model CLIP for object matching, policy learning and
self-termination. A series of experiments indicate that GSP can conduct unknown
object rearrangement with higher completion rate and less steps.