Audio-Infused Automatic Image Colorization by Exploiting Audio Scene
  Semantics
Pengcheng Zhao Yanxiang Chen Yang Zhao Wei Jia Zhao Zhang Ronggang Wang Richang Hong
Automatic image colorization is inherently an ill-posed problem with
uncertainty, which requires an accurate semantic understanding of scenes to
estimate reasonable colors for grayscale images. Although recent
interaction-based methods have achieved impressive performance, it is still a
very difficult task to infer realistic and accurate colors for automatic
colorization. To reduce the difficulty of semantic understanding of grayscale
scenes, this paper tries to utilize corresponding audio, which naturally
contains extra semantic information about the same scene. Specifically, a novel
audio-infused automatic image colorization (AIAIC) network is proposed, which
consists of three stages. First, we take color image semantics as a bridge and
pretrain a colorization network guided by color image semantics. Second, the
natural co-occurrence of audio and video is utilized to learn the color
semantic correlations between audio and visual scenes. Third, the implicit
audio semantic representation is fed into the pretrained network to finally
realize the audio-guided colorization. The whole process is trained in a
self-supervised manner without human annotation. In addition, an audiovisual
colorization dataset is established for training and testing. Experiments
demonstrate that audio guidance can effectively improve the performance of
automatic colorization, especially for some scenes that are difficult to
understand only from visual modality.