Maximum Entropy Model Correction in Reinforcement Learning
Amin Rakhsha Mete Kemertas Mohammad Ghavamzadeh Amir-massoud Farahmand
We propose and theoretically analyze an approach for planning with an
approximate model in reinforcement learning that can reduce the adverse impact
of model error. If the model is accurate enough, it accelerates the convergence
to the true value function too. One of its key components is the MaxEnt Model
Correction (MoCo) procedure that corrects the model's next-state distributions
based on a Maximum Entropy density estimation formulation. Based on MoCo, we
introduce the Model Correcting Value Iteration (MoCoVI) algorithm, and its
sampled-based variant MoCoDyna. We show that MoCoVI and MoCoDyna's convergence
can be much faster than the conventional model-free algorithms. Unlike
traditional model-based algorithms, MoCoVI and MoCoDyna effectively utilize an
approximate model and still converge to the correct value function.