Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI
  Reconstruction
Yunlu Yan Chun-Mei Feng Yuexiang Li Rick Siow Mong Goh Lei Zhu
While multi-modal learning has been widely used for MRI reconstruction, it
relies on paired multi-modal data which is difficult to acquire in real
clinical scenarios. Especially in the federated setting, the common situation
is that several medical institutions only have single-modal data, termed the
modality missing issue. Therefore, it is infeasible to deploy a standard
federated learning framework in such conditions. In this paper, we propose a
novel communication-efficient federated learning framework, namely Fed-PMG, to
address the missing modality challenge in federated multi-modal MRI
reconstruction. Specifically, we utilize a pseudo modality generation mechanism
to recover the missing modality for each single-modal client by sharing the
distribution information of the amplitude spectrum in frequency space. However,
the step of sharing the original amplitude spectrum leads to heavy
communication costs. To reduce the communication cost, we introduce a
clustering scheme to project the set of amplitude spectrum into finite cluster
centroids, and share them among the clients. With such an elaborate design, our
approach can effectively complete the missing modality within an acceptable
communication cost. Extensive experiments demonstrate that our proposed method
can attain similar performance with the ideal scenario, i.e., all clients have
the full set of modalities. The source code will be released.