GePSAn: Generative Procedure Step Anticipation in Cooking Videos
Mohamed Ashraf Abdelsalam Samrudhdhi B. Rangrej Isma Hadji Nikita Dvornik Konstantinos G. Derpanis Afsaneh Fazly
We study the problem of future step anticipation in procedural videos. Given
a video of an ongoing procedural activity, we predict a plausible next
procedure step described in rich natural language. While most previous work
focus on the problem of data scarcity in procedural video datasets, another
core challenge of future anticipation is how to account for multiple plausible
future realizations in natural settings. This problem has been largely
overlooked in previous work. To address this challenge, we frame future step
prediction as modelling the distribution of all possible candidates for the
next step. Specifically, we design a generative model that takes a series of
video clips as input, and generates multiple plausible and diverse candidates
(in natural language) for the next step. Following previous work, we side-step
the video annotation scarcity by pretraining our model on a large text-based
corpus of procedural activities, and then transfer the model to the video
domain. Our experiments, both in textual and video domains, show that our model
captures diversity in the next step prediction and generates multiple plausible
future predictions. Moreover, our model establishes new state-of-the-art
results on YouCookII, where it outperforms existing baselines on the next step
anticipation. Finally, we also show that our model can successfully transfer
from text to the video domain zero-shot, ie, without fine-tuning or adaptation,
and produces good-quality future step predictions from video.