Fisheye Camera and Ultrasonic Sensor Fusion For Near-Field Obstacle
  Perception in Bird's-Eye-View
Arindam Das Sudarshan Paul Niko Scholz Akhilesh Kumar Malviya Ganesh Sistu Ujjwal Bhattacharya Ciarán Eising
Accurate obstacle identification represents a fundamental challenge within
the scope of near-field perception for autonomous driving. Conventionally,
fisheye cameras are frequently employed for comprehensive surround-view
perception, including rear-view obstacle localization. However, the performance
of such cameras can significantly deteriorate in low-light conditions, during
nighttime, or when subjected to intense sun glare. Conversely, cost-effective
sensors like ultrasonic sensors remain largely unaffected under these
conditions. Therefore, we present, to our knowledge, the first end-to-end
multimodal fusion model tailored for efficient obstacle perception in a
bird's-eye-view (BEV) perspective, utilizing fisheye cameras and ultrasonic
sensors. Initially, ResNeXt-50 is employed as a set of unimodal encoders to
extract features specific to each modality. Subsequently, the feature space
associated with the visible spectrum undergoes transformation into BEV. The
fusion of these two modalities is facilitated via concatenation. At the same
time, the ultrasonic spectrum-based unimodal feature maps pass through
content-aware dilated convolution, applied to mitigate the sensor misalignment
between two sensors in the fused feature space. Finally, the fused features are
utilized by a two-stage semantic occupancy decoder to generate grid-wise
predictions for precise obstacle perception. We conduct a systematic
investigation to determine the optimal strategy for multimodal fusion of both
sensors. We provide insights into our dataset creation procedures, annotation
guidelines, and perform a thorough data analysis to ensure adequate coverage of
all scenarios. When applied to our dataset, the experimental results underscore
the robustness and effectiveness of our proposed multimodal fusion approach.