On the Efficacy of Text-Based Input Modalities for Action Anticipation
Apoorva Beedu Karan Samel Irfan Essa
Although the task of anticipating future actions is highly uncertain,
information from additional modalities help to narrow down plausible action
choices. Each modality provides different environmental context for the model
to learn from. While previous multi-modal methods leverage information from
modalities such as video and audio, we primarily explore how text inputs for
actions and objects can also enable more accurate action anticipation.
Therefore, we propose a Multi-modal Anticipative Transformer (MAT), an
attention-based video transformer architecture that jointly learns from
multi-modal features and text captions. We train our model in two-stages, where
the model first learns to predict actions in the video clip by aligning with
captions, and during the second stage, we fine-tune the model to predict future
actions. Compared to existing methods, MAT has the advantage of learning
additional environmental context from two kinds of text inputs: action
descriptions during the pre-training stage, and the text inputs for detected
objects and actions during modality feature fusion. Through extensive
experiments, we evaluate the effectiveness of the pre-training stage, and show
that our model outperforms previous methods on all datasets. In addition, we
examine the impact of object and action information obtained via text and
perform extensive ablations. We evaluate the performance on on three datasets:
EpicKitchens-100, EpicKitchens-55 and EGTEA GAZE+; and show that text
descriptions do indeed aid in more effective action anticipation.