Nighttime Thermal Infrared Image Colorization with Feedback-based Object
  Appearance Learning
Fu-Ya Luo Shu-Lin Liu Yi-Jun Cao Kai-Fu Yang Chang-Yong Xie Yong Liu Yong-Jie Li
Stable imaging in adverse environments (e.g., total darkness) makes thermal
infrared (TIR) cameras a prevalent option for night scene perception. However,
the low contrast and lack of chromaticity of TIR images are detrimental to
human interpretation and subsequent deployment of RGB-based vision algorithms.
Therefore, it makes sense to colorize the nighttime TIR images by translating
them into the corresponding daytime color images (NTIR2DC). Despite the
impressive progress made in the NTIR2DC task, how to improve the translation
performance of small object classes is under-explored. To address this problem,
we propose a generative adversarial network incorporating feedback-based object
appearance learning (FoalGAN). Specifically, an occlusion-aware mixup module
and corresponding appearance consistency loss are proposed to reduce the
context dependence of object translation. As a representative example of small
objects in nighttime street scenes, we illustrate how to enhance the realism of
traffic light by designing a traffic light appearance loss. To further improve
the appearance learning of small objects, we devise a dual feedback learning
strategy to selectively adjust the learning frequency of different samples. In
addition, we provide pixel-level annotation for a subset of the Brno dataset,
which can facilitate the research of NTIR image understanding under multiple
weather conditions. Extensive experiments illustrate that the proposed FoalGAN
is not only effective for appearance learning of small objects, but also
outperforms other image translation methods in terms of semantic preservation
and edge consistency for the NTIR2DC task.