Text-Only Training for Visual Storytelling
Yuechen Wang Wengang Zhou Zhenbo Lu Houqiang Li
Visual storytelling aims to generate a narrative based on a sequence of
images, necessitating both vision-language alignment and coherent story
generation. Most existing solutions predominantly depend on paired image-text
training data, which can be costly to collect and challenging to scale. To
address this, we formulate visual storytelling as a visual-conditioned story
generation problem and propose a text-only training method that separates the
learning of cross-modality alignment and story generation. Our approach
specifically leverages the cross-modality pre-trained CLIP model to integrate
visual control into a story generator, trained exclusively on text data.
Moreover, we devise a training-free visual condition planner that accounts for
the temporal structure of the input image sequence while balancing global and
local visual content. The distinctive advantage of requiring only text data for
training enables our method to learn from external text story data, enhancing
the generalization capability of visual storytelling. We conduct extensive
experiments on the VIST benchmark, showcasing the effectiveness of our approach
in both in-domain and cross-domain settings. Further evaluations on expression
diversity and human assessment underscore the superiority of our method in
terms of informativeness and robustness.