Seeing the Pose in the Pixels: Learning Pose-Aware Representations in
  Vision Transformers
Dominick Reilly Aman Chadha Srijan Das
Human perception of surroundings is often guided by the various poses present
within the environment. Many computer vision tasks, such as human action
recognition and robot imitation learning, rely on pose-based entities like
human skeletons or robotic arms. However, conventional Vision Transformer (ViT)
models uniformly process all patches, neglecting valuable pose priors in input
videos. We argue that incorporating poses into RGB data is advantageous for
learning fine-grained and viewpoint-agnostic representations. Consequently, we
introduce two strategies for learning pose-aware representations in ViTs. The
first method, called Pose-aware Attention Block (PAAB), is a plug-and-play ViT
block that performs localized attention on pose regions within videos. The
second method, dubbed Pose-Aware Auxiliary Task (PAAT), presents an auxiliary
pose prediction task optimized jointly with the primary ViT task. Although
their functionalities differ, both methods succeed in learning pose-aware
representations, enhancing performance in multiple diverse downstream tasks.
Our experiments, conducted across seven datasets, reveal the efficacy of both
pose-aware methods on three video analysis tasks, with PAAT holding a slight
edge over PAAB. Both PAAT and PAAB surpass their respective backbone
Transformers by up to 9.8% in real-world action recognition and 21.8% in
multi-view robotic video alignment. Code is available at
https://github.com/dominickrei/PoseAwareVT.