Towards Viewpoint Robustness in Bird's Eye View Segmentation
Tzofi Klinghoffer Jonah Philion Wenzheng Chen Or Litany Zan Gojcic Jungseock Joo Ramesh Raskar Sanja Fidler Jose M. Alvarez
Autonomous vehicles (AV) require that neural networks used for perception be
robust to different viewpoints if they are to be deployed across many types of
vehicles without the repeated cost of data collection and labeling for each. AV
companies typically focus on collecting data from diverse scenarios and
locations, but not camera rig configurations, due to cost. As a result, only a
small number of rig variations exist across most fleets. In this paper, we
study how AV perception models are affected by changes in camera viewpoint and
propose a way to scale them across vehicle types without repeated data
collection and labeling. Using bird's eye view (BEV) segmentation as a
motivating task, we find through extensive experiments that existing perception
models are surprisingly sensitive to changes in camera viewpoint. When trained
with data from one camera rig, small changes to pitch, yaw, depth, or height of
the camera at inference time lead to large drops in performance. We introduce a
technique for novel view synthesis and use it to transform collected data to
the viewpoint of target rigs, allowing us to train BEV segmentation models for
diverse target rigs without any additional data collection or labeling cost. To
analyze the impact of viewpoint changes, we leverage synthetic data to mitigate
other gaps (content, ISP, etc). Our approach is then trained on real data and
evaluated on synthetic data, enabling evaluation on diverse target rigs. We
release all data for use in future work. Our method is able to recover an
average of 14.7% of the IoU that is otherwise lost when deploying to new rigs.