Perceptual Quality Improvement in Videoconferencing using
  Keyframes-based GAN
Lorenzo Agnolucci Leonardo Galteri Marco Bertini Alberto Del Bimbo
In the latest years, videoconferencing has taken a fundamental role in
interpersonal relations, both for personal and business purposes. Lossy video
compression algorithms are the enabling technology for videoconferencing, as
they reduce the bandwidth required for real-time video streaming. However,
lossy video compression decreases the perceived visual quality. Thus, many
techniques for reducing compression artifacts and improving video visual
quality have been proposed in recent years. In this work, we propose a novel
GAN-based method for compression artifacts reduction in videoconferencing.
Given that, in this context, the speaker is typically in front of the camera
and remains the same for the entire duration of the transmission, we can
maintain a set of reference keyframes of the person from the higher-quality
I-frames that are transmitted within the video stream and exploit them to guide
the visual quality improvement; a novel aspect of this approach is the update
policy that maintains and updates a compact and effective set of reference
keyframes. First, we extract multi-scale features from the compressed and
reference frames. Then, our architecture combines these features in a
progressive manner according to facial landmarks. This allows the restoration
of the high-frequency details lost after the video compression. Experiments
show that the proposed approach improves visual quality and generates
photo-realistic results even with high compression rates. Code and pre-trained
networks are publicly available at
https://github.com/LorenzoAgnolucci/Keyframes-GAN.