3FM: Multi-modal Meta-learning for Federated Tasks
Minh Tran Roochi Shah Zejun Gong
We present a novel approach in the domain of federated learning (FL),
particularly focusing on addressing the challenges posed by modality
heterogeneity, variability in modality availability across clients, and the
prevalent issue of missing data. We introduce a meta-learning framework
specifically designed for multimodal federated tasks. Our approach is motivated
by the need to enable federated models to robustly adapt when exposed to new
modalities, a common scenario in FL where clients often differ in the number of
available modalities. The effectiveness of our proposed framework is
demonstrated through extensive experimentation on an augmented MNIST dataset,
enriched with audio and sign language data. We demonstrate that the proposed
algorithm achieves better performance than the baseline on a subset of missing
modality scenarios with careful tuning of the meta-learning rates. This is a
shortened report, and our work will be extended and updated soon.