Generalizing to Out-of-Sample Degradations via Model Reprogramming
Runhua Jiang Yahong Han
Existing image restoration models are typically designed for specific tasks
and struggle to generalize to out-of-sample degradations not encountered during
training. While zero-shot methods can address this limitation by fine-tuning
model parameters on testing samples, their effectiveness relies on predefined
natural priors and physical models of specific degradations. Nevertheless,
determining out-of-sample degradations faced in real-world scenarios is always
impractical. As a result, it is more desirable to train restoration models with
inherent generalization ability. To this end, this work introduces the
Out-of-Sample Restoration (OSR) task, which aims to develop restoration models
capable of handling out-of-sample degradations. An intuitive solution involves
pre-translating out-of-sample degradations to known degradations of restoration
models. However, directly translating them in the image space could lead to
complex image translation issues. To address this issue, we propose a model
reprogramming framework, which translates out-of-sample degradations by quantum
mechanic and wave functions. Specifically, input images are decoupled as wave
functions of amplitude and phase terms. The translation of out-of-sample
degradation is performed by adapting the phase term. Meanwhile, the image
content is maintained and enhanced in the amplitude term. By taking these two
terms as inputs, restoration models are able to handle out-of-sample
degradations without fine-tuning. Through extensive experiments across multiple
evaluation cases, we demonstrate the effectiveness and flexibility of our
proposed framework. Our codes are available at
\href{https://github.com/ddghjikle/Out-of-sample-restoration}{Github}.