Action Scene Graphs for Long-Form Understanding of Egocentric Videos
Ivan Rodin Antonino Furnari Kyle Min Subarna Tripathi Giovanni Maria Farinella
We present Egocentric Action Scene Graphs (EASGs), a new representation for
long-form understanding of egocentric videos. EASGs extend standard
manually-annotated representations of egocentric videos, such as verb-noun
action labels, by providing a temporally evolving graph-based description of
the actions performed by the camera wearer, including interacted objects, their
relationships, and how actions unfold in time. Through a novel annotation
procedure, we extend the Ego4D dataset by adding manually labeled Egocentric
Action Scene Graphs offering a rich set of annotations designed for long-from
egocentric video understanding. We hence define the EASG generation task and
provide a baseline approach, establishing preliminary benchmarks. Experiments
on two downstream tasks, egocentric action anticipation and egocentric activity
summarization, highlight the effectiveness of EASGs for long-form egocentric
video understanding. We will release the dataset and the code to replicate
experiments and annotations.