PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television
  Dramas
Chen Li Xutan Peng Teng Wang Yixiao Ge Mengyang Liu Xuyuan Xu Yexin Wang Ying Shan
Art forms such as movies and television (TV) dramas are reflections of the
real world, which have attracted much attention from the multimodal learning
community recently. However, existing corpora in this domain share three
limitations: (1) annotated in a scene-oriented fashion, they ignore the
coherence within plots; (2) their text lacks empathy and seldom mentions
situational context; (3) their video clips fail to cover long-form relationship
due to short duration. To address these fundamental issues, using 1,106 TV
drama episodes and 24,875 informative plot-focused sentences written by
professionals, with the help of 449 human annotators, we constructed PTVD, the
first plot-oriented multimodal dataset in the TV domain. It is also the first
non-English dataset of its kind. Additionally, PTVD contains more than 26
million bullet screen comments (BSCs), powering large-scale pre-training. Next,
aiming to open-source a strong baseline for follow-up works, we developed the
multimodal algorithm that attacks different cinema/TV modelling problems with a
unified architecture. Extensive experiments on three cognitive-inspired tasks
yielded a number of novel observations (some of them being quite
counter-intuition), further validating the value of PTVD in promoting
multimodal research. The dataset and codes are released at
\url{https://ptvd.github.io/}.