Unsupervised Text Style Transfer via LLMs and Attention Masking with
  Multi-way Interactions
Lei Pan Yunshi Lan Yang Li Weining Qian
Unsupervised Text Style Transfer (UTST) has emerged as a critical task within
the domain of Natural Language Processing (NLP), aiming to transfer one
stylistic aspect of a sentence into another style without changing its
semantics, syntax, or other attributes. This task is especially challenging
given the intrinsic lack of parallel text pairings. Among existing methods for
UTST tasks, attention masking approach and Large Language Models (LLMs) are
deemed as two pioneering methods. However, they have shortcomings in generating
unsmooth sentences and changing the original contents, respectively. In this
paper, we investigate if we can combine these two methods effectively. We
propose four ways of interactions, that are pipeline framework with tuned
orders; knowledge distillation from LLMs to attention masking model; in-context
learning with constructed parallel examples. We empirically show these
multi-way interactions can improve the baselines in certain perspective of
style strength, content preservation and text fluency. Experiments also
demonstrate that simply conducting prompting followed by attention
masking-based revision can consistently surpass the other systems, including
supervised text style transfer systems. On Yelp-clean and Amazon-clean
datasets, it improves the previously best mean metric by 0.5 and 3.0 absolute
percentages respectively, and achieves new SOTA results.