Finite-Time Error Analysis of Online Model-Based Q-Learning with a
  Relaxed Sampling Model
Han-Dong Lim HyeAnn Lee Donghwan Lee
Reinforcement learning has witnessed significant advancements, particularly
with the emergence of model-based approaches. Among these, $Q$-learning has
proven to be a powerful algorithm in model-free settings. However, the
extension of $Q$-learning to a model-based framework remains relatively
unexplored. In this paper, we delve into the sample complexity of $Q$-learning
when integrated with a model-based approach. Through theoretical analyses and
empirical evaluations, we seek to elucidate the conditions under which
model-based $Q$-learning excels in terms of sample efficiency compared to its
model-free counterpart.