Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting. Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning. As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning. The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise. We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches. In Reinforcement Learning (RL), an agent seeks to maximize the cumulative rewards obtained from interactions with an unknown environment. Given only knowledge based on previously observed trajectories, the agent faces the exploration-exploitation dilemma: Should the agent take actions that maximize rewards based on its current knowledge or instead investigate poorly understood states and actions to potentially improve future performance. Thus, in order to find the optimal policy the agent needs to use an appropriate exploration strategy.Popular exploration strategies, such as -greedy BID37 , rely on random perturbations of the agent's policy, which leads to undirected exploration. The theoretical RL literature offers a variety of statistically-efficient methods that are based on a measure of uncertainty in the agent's model. Examples include upper confidence bound (UCB) BID0 and Thompson sampling (TS) BID40 . In recent years, these have been extended to practical exploration algorithms for large state-spaces and shown to improve performance BID27 O'Donoghue et al., 2018; BID15 . However, these methods assume that the observation noise distribution is independent of the evaluation point, while in practice heteroscedastic observation noise is omnipresent in RL. This means that the noise depends on the evaluation point, rather than being identically distributed (homoscedastic). For instance, the return distribution typically depends on a sequence of interactions and, potentially, on hidden states or inherently heteroscedastic reward observations. BID20 recently demonstrated that, even in the simpler bandit setting, classical approaches such as UCB and TS fail to efficiently account for heteroscedastic noise.In this work, we propose to use Information-Directed Sampling (IDS) BID31 BID20 for efficient exploration in RL. The IDS framework can be used to design exploration-exploitation strategies that balance the estimated instantaneous regret and the expected information gain. Importantly, through the choice of an appropriate information-gain function, IDS is able to account for parametric uncertainty and heteroscedastic observation noise during exploration.As our main contribution, we propose a novel, tractable RL algorithm based on the IDS principle. We combine recent advances in distributional RL BID4 BID12 and approximate parameter uncertainty methods in order to develop both homoscedastic and heteroscedastic variants of an agent that is similar to DQN BID25 , but uses informationdirected exploration. Our evaluation on Atari 2600 games shows the importance of accounting for heteroscedastic noise and indicates that at our approach can substantially outperform alternative state-of-the-art algorithms that focus on modeling either only epistemic or only aleatoric uncertainty. To the best of our knowledge, we are the first to develop a tractable IDS algorithm for RL in large state spaces. We extended the idea of frequentist Information-Directed Sampling to a practical RL exploration algorithm that can account for heteroscedastic noise. To the best of our knowledge, we are the first to propose a tractable IDS algorithm for RL in large state spaces. Our method suggests a new way to use the return distribution in combination with parametric uncertainty for efficient deep exploration and demonstrates substantial gains on Atari games. We also identified several sources of heteroscedasticity in RL and demonstrated the importance of accounting for heteroscedastic noise for efficient exploration. Additionally, our evaluation results demonstrated that similarly to the bandit setting, IDS has the potential to outperform alternative strategies such as TS in RL.There remain promising directions for future work. Our preliminary results show that similar improvements can be observed when IDS is combined with continuous control RL methods such as the Deep Deterministic Policy Gradient (DDPG) BID23 . Developing a computationally efficient approximation of the randomized IDS version, which minimizes the regret-information ratio over the set of stochastic policies, is another idea to investigate. Additionally, as indicated by BID31 , IDS should be seen as a design principle rather than a specific algorithm, and thus alternative information gain functions are an important direction for future research.
We develop a practical extension of Information-Directed Sampling for Reinforcement Learning, which accounts for parametric uncertainty and heteroscedasticity in the return distribution for exploration. The authors propose a way of extending Information-Directed Sampling to reinforcement learning by combining two types of uncertainty to obtain a simple exploration strategy based on IDS.  This paper investigates sophistical exploration approaches for reinforcement learning built on Information Direct Sampling and on Distributional Reinforcement Learning