Large Sequence Models for Sequential Decision-Making: A Survey
Muning Wen Runji Lin Hanjing Wang Yaodong Yang Ying Wen Luo Mai Jun Wang Haifeng Zhang Weinan Zhang
Transformer architectures have facilitated the development of large-scale and
general-purpose sequence models for prediction tasks in natural language
processing and computer vision, e.g., GPT-3 and Swin Transformer. Although
originally designed for prediction problems, it is natural to inquire about
their suitability for sequential decision-making and reinforcement learning
problems, which are typically beset by long-standing issues involving sample
efficiency, credit assignment, and partial observability. In recent years,
sequence models, especially the Transformer, have attracted increasing interest
in the RL communities, spawning numerous approaches with notable effectiveness
and generalizability. This survey presents a comprehensive overview of recent
works aimed at solving sequential decision-making tasks with sequence models
such as the Transformer, by discussing the connection between sequential
decision-making and sequence modeling, and categorizing them based on the way
they utilize the Transformer. Moreover, this paper puts forth various potential
avenues for future research intending to improve the effectiveness of large
sequence models for sequential decision-making, encompassing theoretical
foundations, network architectures, algorithms, and efficient training systems.
As this article has been accepted by the Frontiers of Computer Science, here is
an early version, and the most up-to-date version can be found at
https://journal.hep.com.cn/fcs/EN/10.1007/s11704-023-2689-5