Reason2Drive: Towards Interpretable and Chain-based Reasoning for
  Autonomous Driving
Ming Nie Renyuan Peng Chunwei Wang Xinyue Cai Jianhua Han Hang Xu Li Zhang
Large vision-language models (VLMs) have garnered increasing interest in
autonomous driving areas, due to their advanced capabilities in complex
reasoning tasks essential for highly autonomous vehicle behavior. Despite their
potential, research in autonomous systems is hindered by the lack of datasets
with annotated reasoning chains that explain the decision-making processes in
driving. To bridge this gap, we present Reason2Drive, a benchmark dataset with
over 600K video-text pairs, aimed at facilitating the study of interpretable
reasoning in complex driving environments. We distinctly characterize the
autonomous driving process as a sequential combination of perception,
prediction, and reasoning steps, and the question-answer pairs are
automatically collected from a diverse range of open-source outdoor driving
datasets, including nuScenes, Waymo and ONCE. Moreover, we introduce a novel
aggregated evaluation metric to assess chain-based reasoning performance in
autonomous systems, addressing the semantic ambiguities of existing metrics
such as BLEU and CIDEr. Based on the proposed benchmark, we conduct experiments
to assess various existing VLMs, revealing insights into their reasoning
capabilities. Additionally, we develop an efficient approach to empower VLMs to
leverage object-level perceptual elements in both feature extraction and
prediction, further enhancing their reasoning accuracy. The code and dataset
will be released.