Cross-Modality Proposal-guided Feature Mining for Unregistered
  RGB-Thermal Pedestrian Detection
Chao Tian Zikun Zhou Yuqing Huang Gaojun Li Zhenyu He
RGB-Thermal (RGB-T) pedestrian detection aims to locate the pedestrians in
RGB-T image pairs to exploit the complementation between the two modalities for
improving detection robustness in extreme conditions. Most existing algorithms
assume that the RGB-T image pairs are well registered, while in the real world
they are not aligned ideally due to parallax or different field-of-view of the
cameras. The pedestrians in misaligned image pairs may locate at different
positions in two images, which results in two challenges: 1) how to achieve
inter-modality complementation using spatially misaligned RGB-T pedestrian
patches, and 2) how to recognize the unpaired pedestrians at the boundary. To
deal with these issues, we propose a new paradigm for unregistered RGB-T
pedestrian detection, which predicts two separate pedestrian locations in the
RGB and thermal images, respectively. Specifically, we propose a cross-modality
proposal-guided feature mining (CPFM) mechanism to extract the two precise
fusion features for representing the pedestrian in the two modalities, even if
the RGB-T image pair is unaligned. It enables us to effectively exploit the
complementation between the two modalities. With the CPFM mechanism, we build a
two-stream dense detector; it predicts the two pedestrian locations in the two
modalities based on the corresponding fusion feature mined by the CPFM
mechanism. Besides, we design a data augmentation method, named Homography, to
simulate the discrepancy in scales and views between images. We also
investigate two non-maximum suppression (NMS) methods for post-processing.
Favorable experimental results demonstrate the effectiveness and robustness of
our method in dealing with unregistered pedestrians with different shifts.