Learning Temporal Sentence Grounding From Narrated EgoVideos
Kevin Flanagan Dima Damen Michael Wray
The onset of long-form egocentric datasets such as Ego4D and EPIC-Kitchens
presents a new challenge for the task of Temporal Sentence Grounding (TSG).
Compared to traditional benchmarks on which this task is evaluated, these
datasets offer finer-grained sentences to ground in notably longer videos. In
this paper, we develop an approach for learning to ground sentences in these
datasets using only narrations and their corresponding rough narration
timestamps. We propose to artificially merge clips to train for temporal
grounding in a contrastive manner using text-conditioning attention. This Clip
Merging (CliMer) approach is shown to be effective when compared with a high
performing TSG method -- e.g. mean R@1 improves from 3.9 to 5.7 on Ego4D and
from 10.7 to 13.0 on EPIC-Kitchens. Code and data splits available from:
https://github.com/keflanagan/CliMer