CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening
Wen-Jie Shu Hong-Xia Dou Rui Wen Xiao Wu Liang-Jian Deng
Pansharpening aims to enhance remote sensing image (RSI) quality by merging
high-resolution panchromatic (PAN) with multispectral (MS) images. However,
prior techniques struggled to optimally fuse PAN and MS images for enhanced
spatial and spectral information, due to a lack of a systematic framework
capable of effectively coordinating their individual strengths. In response, we
present the Cross Modulation Transformer (CMT), a pioneering method that
modifies the attention mechanism. This approach utilizes a robust modulation
technique from signal processing, integrating it into the attention mechanism's
calculations. It dynamically tunes the weights of the carrier's value (V)
matrix according to the modulator's features, thus resolving historical
challenges and achieving a seamless integration of spatial and spectral
attributes. Furthermore, considering that RSI exhibits large-scale features and
edge details along with local textures, we crafted a hybrid loss function that
combines Fourier and wavelet transforms to effectively capture these
characteristics, thereby enhancing both spatial and spectral accuracy in
pansharpening. Extensive experiments demonstrate our framework's superior
performance over existing state-of-the-art methods. The code will be publicly
available to encourage further research.