Self-playing Adversarial Language Game Enhances LLM Reasoning
Pengyu Cheng Tianhao Hu Han Xu Zhisong Zhang Yong Dai Lei Han Nan Du
We explore the self-play training procedure of large language models (LLMs)
in a two-player adversarial language game called Adversarial Taboo. In this
game, an attacker and a defender communicate with respect to a target word only
visible to the attacker. The attacker aims to induce the defender to utter the
target word unconsciously, while the defender tries to infer the target word
from the attacker's utterances. To win the game, both players should have
sufficient knowledge about the target word and high-level reasoning ability to
infer and express in this information-reserved conversation. Hence, we are
curious about whether LLMs' reasoning ability can be further enhanced by
Self-Play in this Adversarial language Game (SPAG). With this goal, we let LLMs
act as the attacker and play with a copy of itself as the defender on an
extensive range of target words. Through reinforcement learning on the game
outcomes, we observe that the LLMs' performance uniformly improves on a broad
range of reasoning benchmarks. Furthermore, iteratively adopting this self-play
process can continuously promote LLM's reasoning ability. The code is at
https://github.com/Linear95/SPAG.