PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
Shizhe Chen Ricardo Garcia Cordelia Schmid Ivan Laptev
The ability for robots to comprehend and execute manipulation tasks based on
natural language instructions is a long-term goal in robotics. The dominant
approaches for language-guided manipulation use 2D image representations, which
face difficulties in combining multi-view cameras and inferring precise 3D
positions and relationships. To address these limitations, we propose a 3D
point cloud based policy called PolarNet for language-guided manipulation. It
leverages carefully designed point cloud inputs, efficient point cloud
encoders, and multimodal transformers to learn 3D point cloud representations
and integrate them with language instructions for action prediction. PolarNet
is shown to be effective and data efficient in a variety of experiments
conducted on the RLBench benchmark. It outperforms state-of-the-art 2D and 3D
approaches in both single-task and multi-task learning. It also achieves
promising results on a real robot.