Balanced Representation Learning for Long-tailed Skeleton-based Action
  Recognition
Hongda Liu Yunlong Wang Min Ren Junxing Hu Zhengquan Luo Guangqi Hou Zhenan Sun
Skeleton-based action recognition has recently made significant progress.
However, data imbalance is still a great challenge in real-world scenarios. The
performance of current action recognition algorithms declines sharply when
training data suffers from heavy class imbalance. The imbalanced data actually
degrades the representations learned by these methods and becomes the
bottleneck for action recognition. How to learn unbiased representations from
imbalanced action data is the key to long-tailed action recognition. In this
paper, we propose a novel balanced representation learning method to address
the long-tailed problem in action recognition. Firstly, a spatial-temporal
action exploration strategy is presented to expand the sample space
effectively, generating more valuable samples in a rebalanced manner. Secondly,
we design a detached action-aware learning schedule to further mitigate the
bias in the representation space. The schedule detaches the representation
learning of tail classes from training and proposes an action-aware loss to
impose more effective constraints. Additionally, a skip-modal representation is
proposed to provide complementary structural information. The proposed method
is validated on four skeleton datasets, NTU RGB+D 60, NTU RGB+D 120, NW-UCLA,
and Kinetics. It not only achieves consistently large improvement compared to
the state-of-the-art (SOTA) methods, but also demonstrates a superior
generalization capacity through extensive experiments. Our code is available at
https://github.com/firework8/BRL.