On Offline Evaluation of 3D Object Detection for Autonomous Driving
Tim Schreier Katrin Renz Andreas Geiger Kashyap Chitta
Prior work in 3D object detection evaluates models using offline metrics like
average precision since closed-loop online evaluation on the downstream driving
task is costly. However, it is unclear how indicative offline results are of
driving performance. In this work, we perform the first empirical evaluation
measuring how predictive different detection metrics are of driving performance
when detectors are integrated into a full self-driving stack. We conduct
extensive experiments on urban driving in the CARLA simulator using 16 object
detection models. We find that the nuScenes Detection Score has a higher
correlation to driving performance than the widely used average precision
metric. In addition, our results call for caution on the exclusive reliance on
the emerging class of `planner-centric' metrics.