Design and Analysis of Efficient Attention in Transformers for Social
  Group Activity Recognition
Masato Tamura
Social group activity recognition is a challenging task extended from group
activity recognition, where social groups must be recognized with their
activities and group members. Existing methods tackle this task by leveraging
region features of individuals following existing group activity recognition
methods. However, the effectiveness of region features is susceptible to person
localization and variable semantics of individual actions. To overcome these
issues, we propose leveraging attention modules in transformers to generate
social group features. In this method, multiple embeddings are used to
aggregate features for a social group, each of which is assigned to a group
member without duplication. Due to this non-duplicated assignment, the number
of embeddings must be significant to avoid missing group members and thus
renders attention in transformers ineffective. To find optimal attention
designs with a large number of embeddings, we explore several design choices of
queries for feature aggregation and self-attention modules in transformer
decoders. Extensive experimental results show that the proposed method achieves
state-of-the-art performance and verify that the proposed attention designs are
highly effective on social group activity recognition.