Point Cloud Matters: Rethinking the Impact of Different Observation
  Spaces on Robot Learning
Haoyi Zhu Yating Wang Di Huang Weicai Ye Wanli Ouyang Tong He
In this study, we explore the influence of different observation spaces on
robot learning, focusing on three predominant modalities: RGB, RGB-D, and point
cloud. Through extensive experimentation on over 17 varied contact-rich
manipulation tasks, conducted across two benchmarks and simulators, we have
observed a notable trend: point cloud-based methods, even those with the
simplest designs, frequently surpass their RGB and RGB-D counterparts in
performance. This remains consistent in both scenarios: training from scratch
and utilizing pretraining. Furthermore, our findings indicate that point cloud
observations lead to improved policy zero-shot generalization in relation to
various geometry and visual clues, including camera viewpoints, lighting
conditions, noise levels and background appearance. The outcomes suggest that
3D point cloud is a valuable observation modality for intricate robotic tasks.
We will open-source all our codes and checkpoints, hoping that our insights can
help design more generalizable and robust robotic models.