Object-centric Video Representation for Long-term Action Anticipation
Ce Zhang Changcheng Fu Shijie Wang Nakul Agarwal Kwonjoon Lee Chiho Choi Chen Sun
This paper focuses on building object-centric representations for long-term
action anticipation in videos. Our key motivation is that objects provide
important cues to recognize and predict human-object interactions, especially
when the predictions are longer term, as an observed "background" object could
be used by the human actor in the future. We observe that existing object-based
video recognition frameworks either assume the existence of in-domain
supervised object detectors or follow a fully weakly-supervised pipeline to
infer object locations from action labels. We propose to build object-centric
video representations by leveraging visual-language pretrained models. This is
achieved by "object prompts", an approach to extract task-specific
object-centric representations from general-purpose pretrained models without
finetuning. To recognize and predict human-object interactions, we use a
Transformer-based neural architecture which allows the "retrieval" of relevant
objects for action anticipation at various time scales. We conduct extensive
evaluations on the Ego4D, 50Salads, and EGTEA Gaze+ benchmarks. Both
quantitative and qualitative results confirm the effectiveness of our proposed
method.