Deep Video Inpainting Guided by Audio-Visual Self-Supervision
Kyuyeon Kim Junsik Jung Woo Jae Kim Sung-Eui Yoon
Humans can easily imagine a scene from auditory information based on their
prior knowledge of audio-visual events. In this paper, we mimic this innate
human ability in deep learning models to improve the quality of video
inpainting. To implement the prior knowledge, we first train the audio-visual
network, which learns the correspondence between auditory and visual
information. Then, the audio-visual network is employed as a guider that
conveys the prior knowledge of audio-visual correspondence to the video
inpainting network. This prior knowledge is transferred through our proposed
two novel losses: audio-visual attention loss and audio-visual pseudo-class
consistency loss. These two losses further improve the performance of the video
inpainting by encouraging the inpainting result to have a high correspondence
to its synchronized audio. Experimental results demonstrate that our proposed
method can restore a wider domain of video scenes and is particularly effective
when the sounding object in the scene is partially blinded.