CaptainCook4D: A dataset for understanding errors in procedural
  activities
Rohith Peddi Shivvrat Arya Bharath Challa Likhitha Pallapothula Akshay Vyas Jikai Wang Qifan Zhang Vasundhara Komaragiri Eric Ragan Nicholas Ruozzi Yu Xiang Vibhav Gogate
Following step-by-step procedures is an essential component of various
activities carried out by individuals in their daily lives. These procedures
serve as a guiding framework that helps to achieve goals efficiently, whether
it is assembling furniture or preparing a recipe. However, the complexity and
duration of procedural activities inherently increase the likelihood of making
errors. Understanding such procedural activities from a sequence of frames is a
challenging task that demands an accurate interpretation of visual information
and the ability to reason about the structure of the activity. To this end, we
collect a new egocentric 4D dataset, CaptainCook4D, comprising 384 recordings
(94.5 hours) of people performing recipes in real kitchen environments. This
dataset consists of two distinct types of activity: one in which participants
adhere to the provided recipe instructions and another in which they deviate
and induce errors. We provide 5.3K step annotations and 10K fine-grained action
annotations and benchmark the dataset for the following tasks: supervised error
recognition, multistep localization, and procedure learning