Learned Image Reasoning Prior Penetrates Deep Unfolding Network for
  Panchromatic and Multi-Spectral Image Fusion
Man Zhou Jie Huang Naishan Zheng Chongyi Li
The success of deep neural networks for pan-sharpening is commonly in a form
of black box, lacking transparency and interpretability. To alleviate this
issue, we propose a novel model-driven deep unfolding framework with image
reasoning prior tailored for the pan-sharpening task. Different from existing
unfolding solutions that deliver the proximal operator networks as the
uncertain and vague priors, our framework is motivated by the content reasoning
ability of masked autoencoders (MAE) with insightful designs. Specifically, the
pre-trained MAE with spatial masking strategy, acting as intrinsic reasoning
prior, is embedded into unfolding architecture. Meanwhile, the pre-trained MAE
with spatial-spectral masking strategy is treated as the regularization term
within loss function to constrain the spatial-spectral consistency. Such
designs penetrate the image reasoning prior into deep unfolding networks while
improving its interpretability and representation capability. The uniqueness of
our framework is that the holistic learning process is explicitly integrated
with the inherent physical mechanism underlying the pan-sharpening task.
Extensive experiments on multiple satellite datasets demonstrate the
superiority of our method over the existing state-of-the-art approaches. Code
will be released at \url{https://manman1995.github.io/}.