On the Effects of Fine-tuning Language Models for Text-Based
  Reinforcement Learning
Mauricio Gruppi Soham Dan Keerthiram Murugesan Subhajit Chaudhury
Text-based reinforcement learning involves an agent interacting with a
fictional environment using observed text and admissible actions in natural
language to complete a task. Previous works have shown that agents can succeed
in text-based interactive environments even in the complete absence of semantic
understanding or other linguistic capabilities. The success of these agents in
playing such games suggests that semantic understanding may not be important
for the task. This raises an important question about the benefits of LMs in
guiding the agents through the game states. In this work, we show that rich
semantic understanding leads to efficient training of text-based RL agents.
Moreover, we describe the occurrence of semantic degeneration as a consequence
of inappropriate fine-tuning of language models in text-based reinforcement
learning (TBRL). Specifically, we describe the shift in the semantic
representation of words in the LM, as well as how it affects the performance of
the agent in tasks that are semantically similar to the training games. We
believe these results may help develop better strategies to fine-tune agents in
text-based RL scenarios.