Harnessing the Plug-and-Play Controller by Prompting
Hao Wang Lei Sha
Controllable text generation is a growing field within natural language
generation (NLG) that focuses on producing text that meets specific constraints
in real-world applications. Previous approaches, such as plug-and-play
controllers (PPCs), aimed to steer the properties of generated text in a
flexible manner. However, these methods often compromised the integrity of the
language model's decoding process, resulting in less smooth text generation.
Alternatively, other techniques utilized multiple attribute prompts to align
the generated text with desired attributes, but this approach required prompt
design for each attribute and was dependent on the size of the language model.
This paper introduces a novel method for flexible attribute control in text
generation using pre-trained language models (PLMs). The proposed approach aims
to enhance the fluency of generated text by guiding the generation process with
PPCs. The key idea is to dynamically adjust the distribution of generated text
by modifying prompts, effectively constraining the output space of the language
model and influencing the desired attribute. To enable smooth cooperation
between the PLM and the PPC, our work innovatively proposes a new model
fine-tuning method: Reinforcement Learning with Dynamic Adjust Feedback
(RLDAF).This fine-tuning process adapts a small subset of the language model's
parameters based on the generating actions taken during the PPC control
process. The resulting harmonious collaboration between the PLM and PPC leads
to improved smoothness in text generation during inference. Extensive
experiments were conducted on the SST2 dataset, and the proposed method
outperformed previous approaches in various evaluation metrics, including text
fluency and attribute consistency.