CPST: Comprehension-Preserving Style Transfer for Multi-Modal Narratives
Yi-Chun Chen Arnav Jhala
We investigate the challenges of style transfer in multi-modal visual
narratives. Among static visual narratives such as comics and manga, there are
distinct visual styles in terms of presentation. They include style features
across multiple dimensions, such as panel layout, size, shape, and color. They
include both visual and text media elements. The layout of both text and media
elements is also significant in terms of narrative communication. The
sequential transitions between panels are where readers make inferences about
the narrative world. These feature differences provide an interesting challenge
for style transfer in which there are distinctions between the processing of
features for each modality. We introduce the notion of comprehension-preserving
style transfer (CPST) in such multi-modal domains. CPST requires not only
traditional metrics of style transfer but also metrics of narrative
comprehension. To spur further research in this area, we present an annotated
dataset of comics and manga and an initial set of algorithms that utilize
separate style transfer modules for the visual, textual, and layout parameters.
To test whether the style transfer preserves narrative semantics, we evaluate
this algorithm through visual story cloze tests inspired by work in
computational cognition of narrative systems. Understanding the connection
between style and narrative semantics provides insight for applications ranging
from informational brochure designs to data storytelling.