Language Knowledge-Assisted Representation Learning for Skeleton-Based
  Action Recognition
Haojun Xu Yan Gao Zheng Hui Jie Li Xinbo Gao
How humans understand and recognize the actions of others is a complex
neuroscientific problem that involves a combination of cognitive mechanisms and
neural networks. Research has shown that humans have brain areas that recognize
actions that process top-down attentional information, such as the
temporoparietal association area. Also, humans have brain regions dedicated to
understanding the minds of others and analyzing their intentions, such as the
medial prefrontal cortex of the temporal lobe. Skeleton-based action
recognition creates mappings for the complex connections between the human
skeleton movement patterns and behaviors. Although existing studies encoded
meaningful node relationships and synthesized action representations for
classification with good results, few of them considered incorporating a priori
knowledge to aid potential representation learning for better performance.
LA-GCN proposes a graph convolution network using large-scale language models
(LLM) knowledge assistance. First, the LLM knowledge is mapped into a priori
global relationship (GPR) topology and a priori category relationship (CPR)
topology between nodes. The GPR guides the generation of new "bone"
representations, aiming to emphasize essential node information from the data
level. The CPR mapping simulates category prior knowledge in human brain
regions, encoded by the PC-AC module and used to add additional
supervision-forcing the model to learn class-distinguishable features. In
addition, to improve information transfer efficiency in topology modeling, we
propose multi-hop attention graph convolution. It aggregates each node's
k-order neighbor simultaneously to speed up model convergence. LA-GCN reaches
state-of-the-art on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.