Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in
  Autonomous Driving
Yang Lou Qun Song Qian Xu Rui Tan Jianping Wang
Multi-modal fusion has shown initial promising results for object detection
of autonomous driving perception. However, many existing fusion schemes do not
consider the quality of each fusion input and may suffer from adverse
conditions on one or more sensors. While predictive uncertainty has been
applied to characterize single-modal object detection performance at run time,
incorporating uncertainties into the multi-modal fusion still lacks effective
solutions due primarily to the uncertainty's cross-modal incomparability and
distinct sensitivities to various adverse conditions. To fill this gap, this
paper proposes Uncertainty-Encoded Mixture-of-Experts (UMoE) that explicitly
incorporates single-modal uncertainties into LiDAR-camera fusion. UMoE uses
individual expert network to process each sensor's detection result together
with encoded uncertainty. Then, the expert networks' outputs are analyzed by a
gating network to determine the fusion weights. The proposed UMoE module can be
integrated into any proposal fusion pipeline. Evaluation shows that UMoE
achieves a maximum of 10.67%, 3.17%, and 5.40% performance gain compared with
the state-of-the-art proposal-level multi-modal object detectors under extreme
weather, adversarial, and blinding attack scenarios.