Reinforcement Learning (RL) can model complex behavior policies for goal-directed sequential decision making tasks. A hallmark of RL algorithms is Temporal Difference (TD) learning: value function for the current state is moved towards a bootstrapped target that is estimated using the next state's value function. lambda-returns define the target of the RL agent as a weighted combination of rewards estimated by using multiple many-step look-aheads. Although mathematically tractable, the use of  exponentially decaying weighting of n-step returns based targets in lambda-returns is a rather ad-hoc design choice. Our major contribution  is that we propose a generalization of lambda-returns called Confidence-based Autodidactic Returns (CAR), wherein the RL agent learns the weighting of the n-step returns in an end-to-end manner. In contrast to lambda-returns wherein the RL agent is restricted to use an exponentially decaying weighting scheme, CAR allows the agent to learn to decide how much it wants to weigh the n-step returns based targets. Our experiments, in addition to showing the efficacy of CAR, also empirically demonstrate that using sophisticated weighted mixtures of multi-step returns (like CAR and lambda-returns) considerably outperforms the use of n-step returns. We perform our experiments on the  Asynchronous Advantage Actor Critic (A3C) algorithm in the Atari 2600 domain. Reinforcement Learning (RL) BID21 ) is often used to solve goal-directed sequential decision making tasks wherein conventional Machine Learning methods such as supervised learning are not suitable. Goal-directed sequential decision making tasks are modeled as Markov Decision Process (MDP) BID11 . Traditionally, tabular methods were extensively used for solving MDPs wherein value function or policy estimates were maintained for every state. Such methods become infeasible when the underlying state space of the problem is exponentially large or continuous. Traditional RL methods have also used linear function approximators in conjunction with hand-crafted state spaces for learning policies and value functions. This need for hand-crafted task-specific features has limited the applicability of RL, traditionally.Recent advances in representation learning in the form of deep neural networks provide us with an effective way to achieve generalization BID1 BID6 . Deep neural networks can learn hierarchically compositional representations that enable RL algorithms to generalize over large state spaces. The use of deep neural networks in conjunction with RL objectives has shown remarkable results such as learning to solve the Atari 2600 tasks from raw pixels BID0 BID8 BID16 BID4 , learning to solve complex simulated physics tasks BID24 BID13 BID7 and showing super-human performance on the ancient board game of Go . Building accurate and powerful (in terms of generalization capabilities) state and action value function BID21 estimators is important for successful RL solutions. This is because many practical RL solutions (Q-Learning (Watkins & Dayan, 1992) , SARSA (Rummery & Niranjan, 1994) and Actor-Critic Methods BID5 ) use Temporal Difference (TD) Learning BID20 . In TD learning, a n-step return is used as an estimate of the value function by means of bootstrapping from the n th state's value function estimate. On the other hand, in Monte Carlo learning, the cumulative reward obtained in the entire trajectory following a particular state is used as an estimate for the value function of that state. The ability to build better estimates of the value functions directly results in better policy estimates as well as faster learning. λ-returns (LR) BID21 are very effective in this regard. They are effective for faster propagation of delayed rewards and also result in more reliable learning. LR provide a trade-off between using complete trajectories (Monte Carlo) and bootstrapping from n-step returns (TD learning). They model the TD target using a mixture of n-step returns, wherein the weights of successively longer returns are exponentially decayed. With the advent of deep RL, the use of multi-step returns has gained a lot of popularity BID9 . However, it is to be noted that the use of exponentially decaying weighting for various n-step returns seems to be an ad-hoc design choice made by LR. In this paper, we start off by extensively benchmarking λ-returns (our experiments only use truncated λ-returns due to the nature of the DRL algorithm (A3C) that we work with and we then propose a generalization called the Confidence-based Autodidactic Returns (CAR), In CAR, the DRL agent learns in an end-to-end manner, the weights to assign to the various n-step return based targets. Also in CAR, it's important to note that the weights assigned to various n-step returns change based on the different states from which bootstrapping is done. In this sense, CAR weights are dynamic and using them represents a significant level of sophistication as compared to the usage of λ-returns.In summary, our contributions are:1. To alleviate the need for some ad-hoc choice of weights as in the case of λ-returns, we propose a generalization called Autodidactic Returns and further present a novel derivative of it called Confidence-based Autodidactic Returns (CAR) in the DRL setting.2. We empirically demonstrate that using sophisticated mixtures of multi-step return methods like λ-returns and Confidence-based Autodidactic Returns leads to considerable improvement in the performance of a DRL agent.3. We analyze how the weights learned by CAR are different from that of λ-returns, what the weights signify and how they result in better estimates for the value function. We propose a straightforward way to incorporate λ-returns into the A3C algorithm and carry out a large-scale benchmarking of the resulting algorithm LRA3C. We go on to propose a natural generalization of λ-returns called Confidence-based Autodidactic returns (CAR). In CAR, the agent learns to assign weights dynamically to the various n-step returns from which it can bootstrap. Our experiments demonstrate the efficacy of sophisticated mixture of multi-steps returns with at least one of CARA3C or LRA3C out-performing A3C in 18 out of 22 tasks. In 9 of the tasks CARA3C performs the best whereas in 9 of them LRA3C is the best. CAR gives the agent the freedom to learn and decide how much it wants to weigh each of its n-step returns.The concept of Autodidactic Returns is about the generic idea of giving the DRL agent the ability to model confidence in its own predictions. We demonstrate that this can lead to better Under review as a conference paper at ICLR 2018 TD-targets, in turn leading to improved performances. We have proposed only one way of modeling the autodidactic weights wherein we use the confidence values that are predicted alongside the value function estimates. There are multiple other ways in which these n-step return weights can be modeled. We believe these ways of modeling weighted returns can lead to even better generalization in terms how the agent perceives it's TD-target. Modeling and bootstrapping off TD-targets is fundamental to RL. We believe that our proposed idea of CAR can be combined with any DRL algorithm BID8 BID4 BID16 wherein the TD-target is modeled in terms of n-step returns.
A novel way to generalize lambda-returns by allowing the RL agent to decide how much it wants to weigh each of the n-step returns. Extends the A3C algorithm with lambda returns, and proposes an approach for learning the weights of the returns. The authors present confidence-based autodidactic returns, a Deep learning RL method to adjust the weights of an eligibility vector in TD(lambda)-like value estimation to favour more stable estimates of the state.