Transformers in Reinforcement Learning: A Survey
Pranav Agarwal Aamer Abdul Rahman Pierre-Luc St-Charles Simon J. D. Prince Samira Ebrahimi Kahou
Transformers have significantly impacted domains like natural language
processing, computer vision, and robotics, where they improve performance
compared to other neural networks. This survey explores how transformers are
used in reinforcement learning (RL), where they are seen as a promising
solution for addressing challenges such as unstable training, credit
assignment, lack of interpretability, and partial observability. We begin by
providing a brief domain overview of RL, followed by a discussion on the
challenges of classical RL algorithms. Next, we delve into the properties of
the transformer and its variants and discuss the characteristics that make them
well-suited to address the challenges inherent in RL. We examine the
application of transformers to various aspects of RL, including representation
learning, transition and reward function modeling, and policy optimization. We
also discuss recent research that aims to enhance the interpretability and
efficiency of transformers in RL, using visualization techniques and efficient
training strategies. Often, the transformer architecture must be tailored to
the specific needs of a given application. We present a broad overview of how
transformers have been adapted for several applications, including robotics,
medicine, language modeling, cloud computing, and combinatorial optimization.
We conclude by discussing the limitations of using transformers in RL and
assess their potential for catalyzing future breakthroughs in this field.