Federated Modality-specific Encoders and Multimodal Anchors for
  Personalized Brain Tumor Segmentation
Qian Dai Dong Wei Hong Liu Jinghan Sun Liansheng Wang Yefeng Zheng
Most existing federated learning (FL) methods for medical image analysis only
considered intramodal heterogeneity, limiting their applicability to multimodal
imaging applications. In practice, it is not uncommon that some FL participants
only possess a subset of the complete imaging modalities, posing inter-modal
heterogeneity as a challenge to effectively training a global model on all
participants' data. In addition, each participant would expect to obtain a
personalized model tailored for its local data characteristics from the FL in
such a scenario. In this work, we propose a new FL framework with federated
modality-specific encoders and multimodal anchors (FedMEMA) to simultaneously
address the two concurrent issues. Above all, FedMEMA employs an exclusive
encoder for each modality to account for the inter-modal heterogeneity in the
first place. In the meantime, while the encoders are shared by the
participants, the decoders are personalized to meet individual needs.
Specifically, a server with full-modal data employs a fusion decoder to
aggregate and fuse representations from all modality-specific encoders, thus
bridging the modalities to optimize the encoders via backpropagation reversely.
Meanwhile, multiple anchors are extracted from the fused multimodal
representations and distributed to the clients in addition to the encoder
parameters. On the other end, the clients with incomplete modalities calibrate
their missing-modal representations toward the global full-modal anchors via
scaled dot-product cross-attention, making up the information loss due to
absent modalities while adapting the representations of present ones. FedMEMA
is validated on the BraTS 2020 benchmark for multimodal brain tumor
segmentation. Results show that it outperforms various up-to-date methods for
multimodal and personalized FL and that its novel designs are effective. Our
code is available.