FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology
  Structure and Knowledge Distillation
Jingwen Guo Hong Liu Shitong Sun Tianyu Guo Min Zhang Chenyang Si
Existing skeleton-based action recognition methods typically follow a
centralized learning paradigm, which can pose privacy concerns when exposing
human-related videos. Federated Learning (FL) has attracted much attention due
to its outstanding advantages in privacy-preserving. However, directly applying
FL approaches to skeleton videos suffers from unstable training. In this paper,
we investigate and discover that the heterogeneous human topology graph
structure is the crucial factor hindering training stability. To address this
limitation, we pioneer a novel Federated Skeleton-based Action Recognition
(FSAR) paradigm, which enables the construction of a globally generalized model
without accessing local sensitive data. Specifically, we introduce an Adaptive
Topology Structure (ATS), separating generalization and personalization by
learning a domain-invariant topology shared across clients and a
domain-specific topology decoupled from global model aggregation.Furthermore,
we explore Multi-grain Knowledge Distillation (MKD) to mitigate the discrepancy
between clients and server caused by distinct updating patterns through
aligning shallow block-wise motion features. Extensive experiments on multiple
datasets demonstrate that FSAR outperforms state-of-the-art FL-based methods
while inherently protecting privacy.