Prompt-In-Prompt Learning for Universal Image Restoration
Zilong Li Yiming Lei Chenglong Ma Junping Zhang Hongming Shan
Image restoration, which aims to retrieve and enhance degraded images, is
fundamental across a wide range of applications. While conventional deep
learning approaches have notably improved the image quality across various
tasks, they still suffer from (i) the high storage cost needed for various
task-specific models and (ii) the lack of interactivity and flexibility,
hindering their wider application. Drawing inspiration from the pronounced
success of prompts in both linguistic and visual domains, we propose novel
Prompt-In-Prompt learning for universal image restoration, named PIP. First, we
present two novel prompts, a degradation-aware prompt to encode high-level
degradation knowledge and a basic restoration prompt to provide essential
low-level information. Second, we devise a novel prompt-to-prompt interaction
module to fuse these two prompts into a universal restoration prompt. Third, we
introduce a selective prompt-to-feature interaction module to modulate the
degradation-related feature. By doing so, the resultant PIP works as a
plug-and-play module to enhance existing restoration models for universal image
restoration. Extensive experimental results demonstrate the superior
performance of PIP on multiple restoration tasks, including image denoising,
deraining, dehazing, deblurring, and low-light enhancement. Remarkably, PIP is
interpretable, flexible, efficient, and easy-to-use, showing promising
potential for real-world applications. The code is available at
https://github.com/longzilicart/pip_universal.