Overcoming Distribution Mismatch in Quantizing Image Super-Resolution
  Networks
Cheeun Hong Kyoung Mu Lee
Quantization is a promising approach to reduce the high computational
complexity of image super-resolution (SR) networks. However, compared to
high-level tasks like image classification, low-bit quantization leads to
severe accuracy loss in SR networks. This is because feature distributions of
SR networks are significantly divergent for each channel or input image, and is
thus difficult to determine a quantization range. Existing SR quantization
works approach this distribution mismatch problem by dynamically adapting
quantization ranges to the variant distributions during test time. However,
such dynamic adaptation incurs additional computational costs that limit the
benefits of quantization. Instead, we propose a new quantization-aware training
framework that effectively Overcomes the Distribution Mismatch problem in SR
networks without the need for dynamic adaptation. Intuitively, the mismatch can
be reduced by directly regularizing the variance in features during training.
However, we observe that variance regularization can collide with the
reconstruction loss during training and adversely impact SR accuracy. Thus, we
avoid the conflict between two losses by regularizing the variance only when
the gradients of variance regularization are cooperative with that of
reconstruction. Additionally, to further reduce the distribution mismatch, we
introduce distribution offsets to layers with a significant mismatch, which
either scales or shifts channel-wise features. Our proposed algorithm, called
ODM, effectively reduces the mismatch in distributions with minimal
computational overhead. Experimental results show that ODM effectively
outperforms existing SR quantization approaches with similar or fewer
computations, demonstrating the importance of reducing the distribution
mismatch problem. Our code is available at https://github.com/Cheeun/ODM.