Inclusive Data Representation in Federated Learning: A Novel Approach
  Integrating Textual and Visual Prompt
Zihao Zhao Zhenpeng Shi Yang Liu Wenbo Ding
Federated Learning (FL) is often impeded by communication overhead issues.
Prompt tuning, as a potential solution, has been introduced to only adjust a
few trainable parameters rather than the whole model. However, current
single-modality prompt tuning approaches fail to comprehensively portray local
clients' data. To overcome this limitation, we present Twin Prompt Federated
learning (TPFL), a pioneering solution that integrates both visual and textual
modalities, ensuring a more holistic representation of local clients' data
characteristics. Furthermore, in order to tackle the data heterogeneity issues,
we introduce the Augmented TPFL (ATPFL) employing the contrastive learning to
TPFL, which not only enhances the global knowledge acquisition of client models
but also fosters the development of robust, compact models. The effectiveness
of TPFL and ATPFL is substantiated by our extensive evaluations, consistently
showing superior performance compared to all baselines.