ViLP: Knowledge Exploration using Vision, Language, and Pose Embeddings
  for Video Action Recognition
Soumyabrata Chaudhuri Saumik Bhattacharya
Video Action Recognition (VAR) is a challenging task due to its inherent
complexities. Though different approaches have been explored in the literature,
designing a unified framework to recognize a large number of human actions is
still a challenging problem. Recently, Multi-Modal Learning (MML) has
demonstrated promising results in this domain. In literature, 2D skeleton or
pose modality has often been used for this task, either independently or in
conjunction with the visual information (RGB modality) present in videos.
However, the combination of pose, visual information, and text attributes has
not been explored yet, though text and pose attributes independently have been
proven to be effective in numerous computer vision tasks. In this paper, we
present the first pose augmented Vision-language model (VLM) for VAR. Notably,
our scheme achieves an accuracy of 92.81% and 73.02% on two popular human video
action recognition benchmark datasets, UCF-101 and HMDB-51, respectively, even
without any video data pre-training, and an accuracy of 96.11% and 75.75% after
kinetics pre-training.