STEER: Unified Style Transfer with Expert Reinforcement
Skyler Hallinan Faeze Brahman Ximing Lu Jaehun Jung Sean Welleck Yejin Choi
While text style transfer has many applications across natural language
processing, the core premise of transferring from a single source style is
unrealistic in a real-world setting. In this work, we focus on arbitrary style
transfer: rewriting a text from an arbitrary, unknown style to a target style.
  We propose STEER: Unified Style Transfer with Expert Reinforcement, a unified
frame-work developed to overcome the challenge of limited parallel data for
style transfer. STEER involves automatically generating a corpus of
style-transfer pairs using a product of experts during decoding. The generated
offline data is then used to pre-train an initial policy before switching to
online, off-policy reinforcement learning for further improvements via
fine-grained reward signals. STEER is unified and can transfer to multiple
target styles from an arbitrary, unknown source style, making it particularly
flexible and efficient.
  Experimental results on a challenging dataset with text from a diverse set of
styles demonstrate state-of-the-art results compared to competitive baselines.
Remarkably, STEER outperforms the 175B parameter instruction-tuned GPT-3 on
overall style transfer quality, despite being 226 times smaller in size. We
also show STEER is robust, maintaining its style transfer capabilities on
out-of-domain data, and surpassing nearly all baselines across various styles.
The success of our method highlights the potential of RL algorithms when
augmented with controllable decoding to overcome the challenge of limited data
supervision.