A Dive into SAM Prior in Image Restoration
Zeyu Xiao Jiawang Bai Zhihe Lu Zhiwei Xiong
The goal of image restoration (IR), a fundamental issue in computer vision,
is to restore a high-quality (HQ) image from its degraded low-quality (LQ)
observation. Multiple HQ solutions may correspond to an LQ input in this poorly
posed problem, creating an ambiguous solution space. This motivates the
investigation and incorporation of prior knowledge in order to effectively
constrain the solution space and enhance the quality of the restored images. In
spite of the pervasive use of hand-crafted and learned priors in IR, limited
attention has been paid to the incorporation of knowledge from large-scale
foundation models. In this paper, we for the first time leverage the prior
knowledge of the state-of-the-art segment anything model (SAM) to boost the
performance of existing IR networks in an parameter-efficient tuning manner. In
particular, the choice of SAM is based on its robustness to image degradations,
such that HQ semantic masks can be extracted from it. In order to leverage
semantic priors and enhance restoration quality, we propose a lightweight SAM
prior tuning (SPT) unit. This plug-and-play component allows us to effectively
integrate semantic priors into existing IR networks, resulting in significant
improvements in restoration quality. As the only trainable module in our
method, the SPT unit has the potential to improve both efficiency and
scalability. We demonstrate the effectiveness of the proposed method in
enhancing a variety of methods across multiple tasks, such as image
super-resolution and color image denoising.