Controllable Text Generation with Residual Memory Transformer
Hanqing Zhang Sun Si Haiming Wu Dawei Song
Large-scale Causal Language Models (CLMs), e.g., GPT3 and ChatGPT, have
brought great success in text generation. However, it is still an open
challenge to control the generation process of CLM while balancing flexibility,
control granularity, and generation efficiency. In this paper, we provide a new
alternative for controllable text generation (CTG), by designing a
non-intrusive, lightweight control plugin to accompany the generation of CLM at
arbitrary time steps. The proposed control plugin, namely Residual Memory
Transformer (RMT), has an encoder-decoder setup, which can accept any types of
control conditions and cooperate with CLM through a residual learning paradigm,
to achieve a more flexible, general, and efficient CTG. Extensive experiments
are carried out on various control tasks, in the form of both automatic and
human evaluations. The results show the superiority of RMT over a range of
state-of-the-art approaches, proving the effectiveness and versatility of our
approach.