Energizing Federated Learning via Filter-Aware Attention
Ziyuan Yang Zerui Shao Huijie Huangfu Hui Yu Andrew Beng Jin Teoh Xiaoxiao Li Hongming Shan Yi Zhang
Federated learning (FL) is a promising distributed paradigm, eliminating the
need for data sharing but facing challenges from data heterogeneity.
Personalized parameter generation through a hypernetwork proves effective, yet
existing methods fail to personalize local model structures. This leads to
redundant parameters struggling to adapt to diverse data distributions. To
address these limitations, we propose FedOFA, utilizing personalized orthogonal
filter attention for parameter recalibration. The core is the Two-stream
Filter-aware Attention (TFA) module, meticulously designed to extract
personalized filter-aware attention maps, incorporating Intra-Filter Attention
(IntraFa) and Inter-Filter Attention (InterFA) streams. These streams enhance
representation capability and explore optimal implicit structures for local
models. Orthogonal regularization minimizes redundancy by averting
inter-correlation between filters. Furthermore, we introduce an
Attention-Guided Pruning Strategy (AGPS) for communication efficiency. AGPS
selectively retains crucial neurons while masking redundant ones, reducing
communication costs without performance sacrifice. Importantly, FedOFA operates
on the server side, incurring no additional computational cost on the client,
making it advantageous in communication-constrained scenarios. Extensive
experiments validate superior performance over state-of-the-art approaches,
with code availability upon paper acceptance.