Multi-Semantic Fusion Model for Generalized Zero-Shot Skeleton-Based
  Action Recognition
Ming-Zhe Li Zhen Jia Zhang Zhang Zhanyu Ma Liang Wang
Generalized zero-shot skeleton-based action recognition (GZSSAR) is a new
challenging problem in computer vision community, which requires models to
recognize actions without any training samples. Previous studies only utilize
the action labels of verb phrases as the semantic prototypes for learning the
mapping from skeleton-based actions to a shared semantic space. However, the
limited semantic information of action labels restricts the generalization
ability of skeleton features for recognizing unseen actions. In order to solve
this dilemma, we propose a multi-semantic fusion (MSF) model for improving the
performance of GZSSAR, where two kinds of class-level textual descriptions
(i.e., action descriptions and motion descriptions), are collected as auxiliary
semantic information to enhance the learning efficacy of generalizable skeleton
features. Specially, a pre-trained language encoder takes the action
descriptions, motion descriptions and original class labels as inputs to obtain
rich semantic features for each action class, while a skeleton encoder is
implemented to extract skeleton features. Then, a variational autoencoder (VAE)
based generative module is performed to learn a cross-modal alignment between
skeleton and semantic features. Finally, a classification module is built to
recognize the action categories of input samples, where a seen-unseen
classification gate is adopted to predict whether the sample comes from seen
action classes or not in GZSSAR. The superior performance in comparisons with
previous models validates the effectiveness of the proposed MSF model on
GZSSAR.