Finite-Time Error Analysis of Soft Q-Learning: Switching System Approach
Narim Jeong Donghwan Lee
Soft Q-learning is a variation of Q-learning designed to solve entropy
regularized Markov decision problems where an agent aims to maximize the
entropy regularized value function. Despite its empirical success, there have
been limited theoretical studies of soft Q-learning to date. This paper aims to
offer a novel and unified finite-time, control-theoretic analysis of soft
Q-learning algorithms. We focus on two types of soft Q-learning algorithms: one
utilizing the log-sum-exp operator and the other employing the Boltzmann
operator. By using dynamical switching system models, we derive novel
finite-time error bounds for both soft Q-learning algorithms. We hope that our
analysis will deepen the current understanding of soft Q-learning by
establishing connections with switching system models and may even pave the way
for new frameworks in the finite-time analysis of other reinforcement learning
algorithms.