HiLM-D: Towards High-Resolution Understanding in Multimodal Large
  Language Models for Autonomous Driving
Xinpeng Ding Jianhua Han Hang Xu Wei Zhang Xiaomeng Li
Autonomous driving systems generally employ separate models for different
tasks resulting in intricate designs. For the first time, we leverage singular
multimodal large language models (MLLMs) to consolidate multiple autonomous
driving tasks from videos, i.e., the Risk Object Localization and Intention and
Suggestion Prediction (ROLISP) task. ROLISP uses natural language to
simultaneously identify and interpret risk objects, understand ego-vehicle
intentions, and provide motion suggestions, eliminating the necessity for
task-specific architectures. However, lacking high-resolution (HR) information,
existing MLLMs often miss small objects (e.g., traffic cones) and overly focus
on salient ones (e.g., large trucks) when applied to ROLISP. We propose HiLM-D
(Towards High-Resolution Understanding in MLLMs for Autonomous Driving), an
efficient method to incorporate HR information into MLLMs for the ROLISP task.
Especially, HiLM-D integrates two branches: (i) the low-resolution reasoning
branch, can be any MLLMs, processes low-resolution videos to caption risk
objects and discern ego-vehicle intentions/suggestions; (ii) the
high-resolution perception branch (HR-PB), prominent to HiLM-D,, ingests HR
images to enhance detection by capturing vision-specific HR feature maps and
prioritizing all potential risks over merely salient objects. Our HR-PB serves
as a plug-and-play module, seamlessly fitting into current MLLMs. Experiments
on the ROLISP benchmark reveal HiLM-D's notable advantage over leading MLLMs,
with improvements of 4.8% in BLEU-4 for captioning and 17.2% in mIoU for
detection.