Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging
Jiamian Wang Zongliang Wu Yulun Zhang Xin Yuan Tao Lin Zhiqiang Tao
Snapshot compressive imaging emerges as a promising technology for acquiring
real-world hyperspectral signals. It uses an optical encoder and compressively
produces the 2D measurement, followed by which the 3D hyperspectral data can be
retrieved via training a deep reconstruction network. Existing reconstruction
models are trained with a single hardware instance, whose performance is
vulnerable to hardware perturbation or replacement, demonstrating an
overfitting issue to the physical configuration. This defect limits the
deployment of pre-trained models since they would suffer from large performance
degradation when are assembled to unseen hardware. To better facilitate the
reconstruction model with new hardware, previous efforts resort to centralized
training by collecting multi-hardware and data, which is impractical when
dealing with proprietary assets among institutions. In light of this, federated
learning (FL) has become a feasible solution to enable cross-hardware
cooperation without breaking privacy. However, the naive FedAvg is subject to
client drift upon data heterogeneity owning to the hardware inconsistency. In
this work, we tackle this challenge by marrying prompt tuning with FL to
snapshot compressive imaging for the first time and propose an federated
hardware-prompt learning (FedHP) method. Rather than mitigating the client
drift by rectifying the gradients, which only takes effect on the learning
manifold but fails to touch the heterogeneity rooted in the input data space,
the proposed FedHP globally learns a hardware-conditioned prompter to align the
data distribution, which serves as an indicator of the data inconsistency
stemming from different pre-defined coded apertures. Extensive experiments
demonstrate that the proposed method well coordinates the pre-trained model to
indeterminate hardware configurations.