Event-Guided Procedure Planning from Instructional Videos with Text
  Supervision
An-Lan Wang Kun-Yu Lin Jia-Run Du Jingke Meng Wei-Shi Zheng
In this work, we focus on the task of procedure planning from instructional
videos with text supervision, where a model aims to predict an action sequence
to transform the initial visual state into the goal visual state. A critical
challenge of this task is the large semantic gap between observed visual states
and unobserved intermediate actions, which is ignored by previous works.
Specifically, this semantic gap refers to that the contents in the observed
visual states are semantically different from the elements of some action text
labels in a procedure. To bridge this semantic gap, we propose a novel
event-guided paradigm, which first infers events from the observed states and
then plans out actions based on both the states and predicted events. Our
inspiration comes from that planning a procedure from an instructional video is
to complete a specific event and a specific event usually involves specific
actions. Based on the proposed paradigm, we contribute an Event-guided
Prompting-based Procedure Planning (E3P) model, which encodes event information
into the sequential modeling process to support procedure planning. To further
consider the strong action associations within each event, our E3P adopts a
mask-and-predict approach for relation mining, incorporating a probabilistic
masking scheme for regularization. Extensive experiments on three datasets
demonstrate the effectiveness of our proposed model.