Temporal Difference learning with function approximation has been widely used recently and has led to several successful results.   However, compared with the original tabular-based methods, one major drawback of temporal difference learning with neural networks and other function approximators is that they tend to over-generalize across temporally successive states, resulting in slow convergence and even instability. In this work, we propose a novel TD learning method, Hadamard product Regularized TD (HR-TD), that reduces over-generalization and thus leads to faster convergence. This approach can be easily applied to both linear and nonlinear function approximators. 
 HR-TD is evaluated on several linear and nonlinear benchmark domains, where we show improvement in learning behavior and performance. Temporal Difference (TD) learning is one of the most important paradigms in Reinforcement Learning BID15 . Techniques based on combining TD learning with nonlinear function approximators and stochastic gradient descent, such as deep networks, have led to significant breakthroughs in large-scale problems to which these methods can be applied BID8 BID12 .At its heart, the TD learning update is straightforward. v(s ) estimates the value of being in a state s. After an action a that transitions the agent from s to next state s , v(s) is altered to be closer to the (discounted) estimated value of s , v(s ) (plus any received reward, r). The difference between these estimated values is called the temporal difference error (TD error) and is typically denoted as δ. Formally, δ = r + γv(s ) − v(s), where γ is the discount factor, and r + γv(s ) is known as the TD target.When states are represented individually (the tabular case), v(s) can be altered independently from v(s ) using the update rule v(s) ← v(s) + αδ, where α is the learning rate. In fully deterministic environments , α can be set to 1, thus causing v(s) to change all the way to the TD target. Otherwise, in a stochastic environment , α is set less than 1 so that v(s) only moves part of the way towards the TD target, thus avoiding over-generalization from a single example. When, on the other hand, states are represented with a function approximator, as is necessary in large or continuous environments, v(s) can no longer be updated independently from v(s ). That is because s and s are likely to be similar (assuming actions have local effects), any change to v(s) is likely to also alter v(s ). While such generalization is desirable in principle, it also has the unintended consequence of changing the TD target, which in turn can cause the TD update to lead to an increase in the TD error between s and s . This unintended consequence can be seen as a second form of over-generalization: one that can be much more difficult to avoid.Past work has identified this form of over-generalization in RL, has observed that it is particularly relevant in methods that use neural network function approximators such as DQN BID8 , and has proposed initial solutions BID4 BID10 . In this paper, we present a deeper analysis of the reasons for this form of over-generalization and introduce a novel learning algorithm termed HR-TD, based on the recursive proximal mapping formulation of TD learning BID1 , which offers a mathematical framework for parameter regularization that allows one to control for this form of over-generalization. Empirical results across multiple domains demonstrate that our novel algorithm learns more efficiently (from fewer samples) than prior approaches.The rest of the paper is organized as follows. Section 2 offers a brief background on TD learning, the over-generalization problem, and optimization techniques used in the derivation of our algorithm. In Section 3, we discuss the state-of-the-art research in this direction. The motivation and the design of our algorithm are presented in Section 4. Finally , the experimental results of Section 5 validate the effectiveness of the proposed algorithm. In this paper, we analyze the problem of over-generalization in TD learning with function approximation. This analysis points to the potential pitfalls of over-generalization in TD-learning. Based on the analysis, we propose a novel regularization scheme based on the Hadamard product. We also show that with the right weight on the regularization, the solution of this method is the same as that of TD. Finally, we experimentally validate the effectiveness of our algorithm on benchmarks of varying complexity.
A regularization technique for TD learning that avoids temporal over-generalization, especially in Deep Networks A variation on temporal difference learning for the function approximation case that attempts to resolve the issue of over-generalization across temporally-successive states. The paper introduces HR-TD, a variation of the TD(0) algorithm, meant to improve the over-generalization problem in conventional TD