Efficient Diffusion Model for Image Restoration by Residual Shifting
Zongsheng Yue Jianyi Wang Chen Change Loy
While diffusion-based image restoration (IR) methods have achieved remarkable
success, they are still limited by the low inference speed attributed to the
necessity of executing hundreds or even thousands of sampling steps. Existing
acceleration sampling techniques, though seeking to expedite the process,
inevitably sacrifice performance to some extent, resulting in over-blurry
restored outcomes. To address this issue, this study proposes a novel and
efficient diffusion model for IR that significantly reduces the required number
of diffusion steps. Our method avoids the need for post-acceleration during
inference, thereby avoiding the associated performance deterioration.
Specifically, our proposed method establishes a Markov chain that facilitates
the transitions between the high-quality and low-quality images by shifting
their residuals, substantially improving the transition efficiency. A carefully
formulated noise schedule is devised to flexibly control the shifting speed and
the noise strength during the diffusion process. Extensive experimental
evaluations demonstrate that the proposed method achieves superior or
comparable performance to current state-of-the-art methods on three classical
IR tasks, namely image super-resolution, image inpainting, and blind face
restoration, \textit{\textbf{even only with four sampling steps}}. Our code and
model are publicly available at \url{https://github.com/zsyOAOA/ResShift}.