Cross-Attention Watermarking of Large Language Models
Folco Bertini Baldassini Huy H. Nguyen Ching-Chung Chang Isao Echizen
A new approach to linguistic watermarking of language models is presented in
which information is imperceptibly inserted into the output text while
preserving its readability and original meaning. A cross-attention mechanism is
used to embed watermarks in the text during inference. Two methods using
cross-attention are presented that minimize the effect of watermarking on the
performance of a pretrained model. Exploration of different training strategies
for optimizing the watermarking and of the challenges and implications of
applying this approach in real-world scenarios clarified the tradeoff between
watermark robustness and text quality. Watermark selection substantially
affects the generated output for high entropy sentences. This proactive
watermarking approach has potential application in future model development.