CausalChaos! Dataset for Comprehensive Causal Action Question Answering
  Over Longer Causal Chains Grounded in Dynamic Visual Scenes
Ting En Lam Yuhan Chen Elston Tan Eric Peh Ruirui Chen Paritosh Parmar Basura Fernando
Causal video question answering (QA) has garnered increasing interest, yet
existing datasets often lack depth in causal reasoning analysis. To address
this gap, we capitalize on the unique properties of cartoons and construct
CausalChaos!, a novel, challenging causal Why-QA dataset built upon the iconic
"Tom and Jerry" cartoon series. With thoughtful questions and multi-level
answers, our dataset contains much longer causal chains embedded in dynamic
interactions and visuals, at the same time principles of animation allows
animators to create well-defined, unambiguous causal relationships. These
factors allow models to solve more challenging, yet well-defined causal
relationships. We also introduce hard negative mining, including
CausalConfusion version. While models perform well, there is much room for
improvement, especially, on open-ended answers. We identify more
advanced/explicit causal relationship modeling and joint modeling of vision and
language as the immediate areas for future efforts to focus upon. Along with
the other complementary datasets, our new challenging dataset will pave the way
for these developments in the field. We will release our dataset, codes, and
models to help future efforts in this domain.