Masked Diffusion with Task-awareness for Procedure Planning in
  Instructional Videos
Fen Fang Yun Liu Ali Koksal Qianli Xu Joo-Hwee Lim
A key challenge with procedure planning in instructional videos lies in how
to handle a large decision space consisting of a multitude of action types that
belong to various tasks. To understand real-world video content, an AI agent
must proficiently discern these action types (e.g., pour milk, pour water, open
lid, close lid, etc.) based on brief visual observation. Moreover, it must
adeptly capture the intricate semantic relation of the action types and task
goals, along with the variable action sequences. Recently, notable progress has
been made via the integration of diffusion models and visual representation
learning to address the challenge. However, existing models employ rudimentary
mechanisms to utilize task information to manage the decision space. To
overcome this limitation, we introduce a simple yet effective enhancement - a
masked diffusion model. The introduced mask acts akin to a task-oriented
attention filter, enabling the diffusion/denoising process to concentrate on a
subset of action types. Furthermore, to bolster the accuracy of task
classification, we harness more potent visual representation learning
techniques. In particular, we learn a joint visual-text embedding, where a text
embedding is generated by prompting a pre-trained vision-language model to
focus on human actions. We evaluate the method on three public datasets and
achieve state-of-the-art performance on multiple metrics. Code is available at
https://github.com/ffzzy840304/Masked-PDPP.