PRED: Pre-training via Semantic Rendering on LiDAR Point Clouds
Hao Yang Haiyang Wang Di Dai Liwei Wang
Pre-training is crucial in 3D-related fields such as autonomous driving where
point cloud annotation is costly and challenging. Many recent studies on point
cloud pre-training, however, have overlooked the issue of incompleteness, where
only a fraction of the points are captured by LiDAR, leading to ambiguity
during the training phase. On the other hand, images offer more comprehensive
information and richer semantics that can bolster point cloud encoders in
addressing the incompleteness issue inherent in point clouds. Yet,
incorporating images into point cloud pre-training presents its own challenges
due to occlusions, potentially causing misalignments between points and pixels.
In this work, we propose PRED, a novel image-assisted pre-training framework
for outdoor point clouds in an occlusion-aware manner. The main ingredient of
our framework is a Birds-Eye-View (BEV) feature map conditioned semantic
rendering, leveraging the semantics of images for supervision through neural
rendering. We further enhance our model's performance by incorporating
point-wise masking with a high mask ratio (95%). Extensive experiments
demonstrate PRED's superiority over prior point cloud pre-training methods,
providing significant improvements on various large-scale datasets for 3D
perception tasks. Codes will be available at https://github.com/PRED4pc/PRED.