Learning from a scalar reward in continuous action space environments is difficult and often requires millions if not billions of interactions.   We introduce state aligned vector rewards, which are easily defined in metric state spaces and allow our deep reinforcement learning agent to tackle the curse of dimensionality.   Our agent learns to map from action distributions to state change distributions implicitly defined in a quantile function neural network.    We further introduce a new reinforcement learning technique inspired by quantile regression which does not limit agents to explicitly parameterized action distributions.   Our results in high dimensional state spaces show that training with vector rewards allows our agent to learn multiple times faster than an agent training with scalar rewards. Reinforcement learning BID32 ) is a powerful paradigm in which an agent learns about an environment through interaction. The common formulation consists of a Markov Decision Process (MDP) modeled as a 5-tuple (S, A, P, r, γ) where S is the (possibly infinite) set of states, A is the (possibly infinite) set of actions available to the agent, P : (S × A × S) → [0, 1] : P (s |s, a) is the transition probability of reaching state s ∈ S given state s ∈ S and action a ∈ A, r : (S × A) → R : r(s, a) is the reward received for taking action a in state s and γ is the reward discount factor. The goal of the agent is to maximize the cumulative discounted reward R = ∞ t=0 γ t r(s t , a t ) by choosing actions a t according to some (possibly stochastic) policy π : (S × A) → [0, 1] : π(a t |s t ). Sometimes it is further useful to make a distinction between the actual state space S and the correlated observation space O of the agent. In this case π : (O × A) → [0, 1] : π(a t |o t ) with o t ∈ O. The use of deep neural networks allowed this formulation to scale to high dimensional visual inputs approaching continuity in state space BID20 while others extended deep reinforcement learning to continuous action spaces BID15 BID21 . While neural networks are powerful function approximators, they require large amounts of training data to converge. In the case of reinforcement learning this means interactions with the environment, a requirement easy to fulfill in simulation, yet impractical when the agent should interact with the real world. This problem is aggravated by the weak training signal of classical reinforcement learning -a simple scalar reward.While originally the dopamine activity in mammal brains was linked to general "rewarding" events, BID28 point out that the diversity of dopamine circuits in the mid brain is better modeled by viral vector strategies. BID8 also show that human reinforcement learning incorporates effector specific value estimations to cope with the high dimensional action space. Inspired by these biological insights, we improve the sample efficiency of a deep reinforcement learning algorithm in this work by modeling a d-dimensional vector reward. A vector reward can in some domains easily be defined in alignment with the state space. We say that two vector spaces are aligned if their dimensions correlate and show that if state and action space are not aligned, a mapping from action distribution to state change distribution can be learned.As a motivating example, consider the agent in Figure 1 (a) trying to reach the goal (marked by the blue dot). If we take p as the position vector of the agent relative to the goal, a sensible reward to guide the agent to the goal in this environment would be r = ||p|| − ||p + a|| where || · || can be any norm in the vector space of the environment. For illustration purposes we'll focus on the L 1 norm in this example and throughout this paper. During training the agent might try action a which moves Figure 1: (a) An agent freely moving in a 2D world might try to reach a goal at position (0, 0) by taking action a. A sensible reward in this environment is the change in absolute distance to the goal. With a scalar reward this would be summarized as r = r x − r y , whereas a vector reward would keep the two reward dimensions distinguishable. (b) In most cases action and state space are however not aligned, therefore a mapping from action to state change must be learned.it closer to the goal in x direction, but a bit further away from the goal in y direction. The scalar reward would then just convey the information, that the action was rather positive (since the agent got closer to the goal) but miss out on the distinction that the action was good in x-direction but bad in y-direction. To provide this distinction, a more informative reward would keep the dimensions separate and therefore be a vector itself: r = |p| − |p + a| where | · | denotes the element-wise absolute value here. Note that this reward is dimension wise aligned with the position p, the state, of the agent. Since we focus on reaching problems in this work, we'll use the terms "position" and "state" interchangeably.The problem with such a state aligned vector reward is however that the action space is in most cases not state aligned. To see this, consider the schematic robot arm in Figure 1 (b): The action dimensions a 1 and a 2 correspond to the torques of the robot arm and do not directly translate to a shift in x and y dimension, respectively. To address this issue we use the method proposed by BID5 a) to train a deep neural network to approximate the quantile function, in our case of the position change, given the current observation and quantile target. Additionally we give a parameterization of the action probability distribution as input to this position change prediction network (short PCPN). We then train the agent, parameterized by another neural network which maps from observations to action probability distributions, through a new reinforcement learning method we call quantile regression reinforcement learning (short QRRL). A schematic overview of our setup can be seen in FIG1 .To summarize, the contributions of this paper are the following:• We extend the reinforcement learning paradigm to allow for faster training based on more informative state aligned vector rewards.• We present an architecture that learns a probability distribution over possible state changes based on a probability distribution over possible actions.• We introduce a new reinforcement learning algorithm to train stochastic continuous action policies with arbitrary action probability distributions. In this work we present the idea of state aligned vector rewards for faster reinforcement learning. While the idea is straight forward and simple, we are unaware of any work that addresses it so far. Additionally, we also present a new reinforcement learning technique based on quantile regression in this work which we term QRRL. QRRL allows for complex stochastic policies in continuous action spaces, not limiting agents to Gaussian actions. Combining both, we show that the agent network in our SAVER agent can be trained through a quantile network pretrained in the environment. We show that SAVER is capable of training orders of magnitudes faster in high dimensional metric spaces. While d-dimensional metric spaces are mainly mathematical constructs for d > 3, we see a lot of potential in SAVER to be applied to problems in mathematics and related fields, including the field of deep (reinforcement) learning itself.
We train with state aligned vector rewards an agent predicting state changes from action distributions, using a new reinforcement learning technique inspired by quantile regression. Presents algorithm that aims to speed up reinforcement learning in situations where the reward is aligned with the state space.  This paper addresses RL in the continuous action space, using a re-parametrised policy and a novel vector-based training objective. This work proposes to mix distributional RL with a net in charge of modeling the evolution of the world in terms of quantiles, claiming improvements in sample efficiency.