Vision meets mmWave Radar: 3D Object Perception Benchmark for Autonomous
  Driving
Yizhou Wang Jen-Hao Cheng Jui-Te Huang Sheng-Yao Kuan Qiqian Fu Chiming Ni Shengyu Hao Gaoang Wang Guanbin Xing Hui Liu Jenq-Neng Hwang
Sensor fusion is crucial for an accurate and robust perception system on
autonomous vehicles. Most existing datasets and perception solutions focus on
fusing cameras and LiDAR. However, the collaboration between camera and radar
is significantly under-exploited. The incorporation of rich semantic
information from the camera, and reliable 3D information from the radar can
potentially achieve an efficient, cheap, and portable solution for 3D object
perception tasks. It can also be robust to different lighting or all-weather
driving scenarios due to the capability of mmWave radars. In this paper, we
introduce the CRUW3D dataset, including 66K synchronized and well-calibrated
camera, radar, and LiDAR frames in various driving scenarios. Unlike other
large-scale autonomous driving datasets, our radar data is in the format of
radio frequency (RF) tensors that contain not only 3D location information but
also spatio-temporal semantic information. This kind of radar format can enable
machine learning models to generate more reliable object perception results
after interacting and fusing the information or features between the camera and
radar.