Fast Value Tracking for Deep Reinforcement Learning
Frank Shih Faming Liang
Reinforcement learning (RL) tackles sequential decision-making problems by
creating agents that interacts with their environment. However, existing
algorithms often view these problem as static, focusing on point estimates for
model parameters to maximize expected rewards, neglecting the stochastic
dynamics of agent-environment interactions and the critical role of uncertainty
quantification. Our research leverages the Kalman filtering paradigm to
introduce a novel and scalable sampling algorithm called Langevinized Kalman
Temporal-Difference (LKTD) for deep reinforcement learning. This algorithm,
grounded in Stochastic Gradient Markov Chain Monte Carlo (SGMCMC), efficiently
draws samples from the posterior distribution of deep neural network
parameters. Under mild conditions, we prove that the posterior samples
generated by the LKTD algorithm converge to a stationary distribution. This
convergence not only enables us to quantify uncertainties associated with the
value function and model parameters but also allows us to monitor these
uncertainties during policy updates throughout the training phase. The LKTD
algorithm paves the way for more robust and adaptable reinforcement learning
approaches.