Enhancing Traffic Object Detection in Variable Illumination with
  RGB-Event Fusion
Zhanwen Liu Nan Yang Yang Wang Yuke Li Xiangmo Zhao Fei-Yue Wang
Traffic object detection under variable illumination is challenging due to
the information loss caused by the limited dynamic range of conventional
frame-based cameras. To address this issue, we introduce bio-inspired event
cameras and propose a novel Structure-aware Fusion Network (SFNet) that
extracts sharp and complete object structures from the event stream to
compensate for the lost information in images through cross-modality fusion,
enabling the network to obtain illumination-robust representations for traffic
object detection. Specifically, to mitigate the sparsity or blurriness issues
arising from diverse motion states of traffic objects in fixed-interval event
sampling methods, we propose the Reliable Structure Generation Network (RSGNet)
to generate Speed Invariant Frames (SIF), ensuring the integrity and sharpness
of object structures. Next, we design a novel Adaptive Feature Complement
Module (AFCM) which guides the adaptive fusion of two modality features to
compensate for the information loss in the images by perceiving the global
lightness distribution of the images, thereby generating illumination-robust
representations. Finally, considering the lack of large-scale and high-quality
annotations in the existing event-based object detection datasets, we build a
DSEC-Det dataset, which consists of 53 sequences with 63,931 images and more
than 208,000 labels for 8 classes. Extensive experimental results demonstrate
that our proposed SFNet can overcome the perceptual boundaries of conventional
cameras and outperform the frame-based method by 8.0% in mAP50 and 5.9% in
mAP50:95. Our code and dataset will be available at
https://github.com/YN-Yang/SFNet.