We propose Bayesian Deep Q-Network  (BDQN), a  practical Thompson sampling based Reinforcement Learning (RL) Algorithm. Thompson sampling allows for targeted exploration in high dimensions through posterior sampling but is usually computationally expensive. We address this limitation by introducing uncertainty only at the output layer of the network through a Bayesian Linear Regression (BLR) model, which can be trained with fast closed-form updates and its samples can be drawn efficiently through the Gaussian distribution. We apply our method to a wide range of Atari Arcade Learning Environments. Since BDQN carries out more efficient exploration, it is able to reach higher rewards substantially faster than a key baseline, DDQN. Designing algorithms that achieve an optimal trade-off between exploration and exploitation is one of the primary goal of reinforcement learning (RL). However, targeted exploration in high dimensional spaces is a challenging problem in RL. Recent advances in deep RL mostly deploy simple exploration strategies such as ε-greedy, where the agent chooses the optimistic action, the action with highest promising return, with probability (1 − ε), otherwise, uniformly at random picks one of the available actions. Due to this uniform sampling, the ε-greedy method scales poorly with the dimensionality of state and action spaces. Recent work has considered scaling exploration strategies to large domains BID12 . Several of these papers have focused on employing optimism-under-uncertainty approaches, which essentially rely on computing confidence bounds over different actions, and acting optimistically with respect to that uncertainty.An alternative to optimism-under-uncertainty (Brafman & Tennenholtz, 2003) is Thompson Sampling (TS) BID57 , one of the oldest heuristics for multi arm bandits. TS is a Bayesian approach where one starts with a prior distribution over the belief and compute the posterior beliefs based on the collected data through the interaction with the environment and then maximizes the expected return under the sampled belief. The TS based posterior sampling can provide more targeted exploration since it can trade off uncertainty with the expected return of actions. In contrast, the ε-greedy strategy is indifferent to uncertainty of the actions and the expected rewards of sub-optimistic ones (set of actions excluding the optimistic action).There has been relatively little work on scaling Thompson Sampling to large state spaces. The primary difficulty in implementing Thompson sampling is the difficulty of sampling from general posterior distributions. Prior efforts in this space have generally required extremely expensive computations (e.g. BID21 BID52 )We derive a practical Thompson sampling framework, termed as Bayesian deep Q-networks (BDQN), where we approximate the posterior distribution on the set of Q-functions and sample from this approximated posterior. BDQN is computationally efficient since it incorporates uncertainty only at the output layer, in the form of a Bayesian linear regression model. Due to linearity and by choosing a Gaussian prior, we derive a closed-form analytical update to the approximated posterior distribution over Q functions. We can also draw samples efficiently from the Gaussian distribution. As addressed in BID32 , one of the major benefits of function approximation methods in deep RL is that the estimation of the Q-value, given a state-action pair, can generalize well to other state-action pairs, even if they are visited rarely. We expect this to hold in BDQN as well, but additionally, we also expect the uncertainty of state-action pairs to generalize well.We test BDQN on a wide range of Arcade Learning Environment Atari games BID13 BID15 ) against a strong baseline DDQN BID58 . Aside from simplicity and popularity of DDQN, BDQN and DDQN share the same architecture, and follow same target objective. These are the main reasons we choose DDQN for our comparisons.In table. 1 we see significant gains for BDQN over DDQN. BDQN is able to learn significantly faster and reach higher returns due to more efficient exploration. The evidence of this is further seen from the fact that we are able to train BDQN with much higher learning rates compared to DDQN. This suggests that BDQN is able to learn faster and reach better scores.These promising results suggest that BDQN can further benefit from additional modifications that were done to DQN, e.g. Prioritized Experience Replay , Dueling approach , A3C BID33 , safe exploration BID30 , and etc. This is because BDQN only changes that exploration strategy of DQN, and can easily accommodate additional improvements to DQN.
Using Bayesian regression to estimate the posterior over Q-functions and deploy Thompson Sampling as a targeted exploration strategy with efficient trade-off the exploration and exploitation The authors propose a new algorithm for exploration in Deep RL where they apply Bayesian linear regression with features from the last layer of a DQN network to estimate the Q function for each action. The authors describe how to use Bayesian neural networks with Thompson sampling for efficient exploration in q-learning and propose an approach that outperforms epsilon-greedy exploration approaches.