Do You Remember? Dense Video Captioning with Cross-Modal Memory
  Retrieval
Minkuk Kim Hyeon Bae Kim Jinyoung Moon Jinwoo Choi Seong Tae Kim
There has been significant attention to the research on dense video
captioning, which aims to automatically localize and caption all events within
untrimmed video. Several studies introduce methods by designing dense video
captioning as a multitasking problem of event localization and event captioning
to consider inter-task relations. However, addressing both tasks using only
visual input is challenging due to the lack of semantic content. In this study,
we address this by proposing a novel framework inspired by the cognitive
information processing of humans. Our model utilizes external memory to
incorporate prior knowledge. The memory retrieval method is proposed with
cross-modal video-to-text matching. To effectively incorporate retrieved text
features, the versatile encoder and the decoder with visual and textual
cross-attention modules are designed. Comparative experiments have been
conducted to show the effectiveness of the proposed method on ActivityNet
Captions and YouCook2 datasets. Experimental results show promising performance
of our model without extensive pretraining from a large video dataset.