SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional
  Videos
Yulei Niu Wenliang Guo Long Chen Xudong Lin Shih-Fu Chang
We study the problem of procedure planning in instructional videos, which
aims to make a goal-oriented sequence of action steps given partial visual
state observations. The motivation of this problem is to learn a structured and
plannable state and action space. Recent works succeeded in sequence modeling
of steps with only sequence-level annotations accessible during training, which
overlooked the roles of states in the procedures. In this work, we point out
that State CHangEs MAtter (SCHEMA) for procedure planning in instructional
videos. We aim to establish a more structured state space by investigating the
causal relations between steps and states in procedures. Specifically, we
explicitly represent each step as state changes and track the state changes in
procedures. For step representation, we leveraged the commonsense knowledge in
large language models (LLMs) to describe the state changes of steps via our
designed chain-of-thought prompting. For state change tracking, we align visual
state observations with language state descriptions via cross-modal contrastive
learning, and explicitly model the intermediate states of the procedure using
LLM-generated state descriptions. Experiments on CrossTask, COIN, and NIV
benchmark datasets demonstrate that our proposed SCHEMA model achieves
state-of-the-art performance and obtains explainable visualizations.