We introduce a new approach to estimate continuous actions using actor-critic algorithms for reinforcement learning problems. Policy gradient methods usually predict one continuous action estimate or parameters of a presumed distribution (most commonly Gaussian) for any given state which might not be optimal as it may not capture the complete description of the target distribution. Our approach instead predicts M actions with the policy network (actor) and then uniformly sample one action during training as well as testing at each state. This allows the agent to learn a simple stochastic policy that has an easy to compute expected return. In all experiments, this facilitates better exploration of the state space during training and converges to a better policy. Reinforcement learning is a traditional branch of machine learning which focuses on learning complex tasks by assigning rewards to agents that interact with their environment. It has recently gained momentum thanks to the combination of novel algorithms for continuous control with deep learning models, sometimes even matching human performance in tasks such as playing video games and manipulating objects BID10 ; . Recent methods for continuous control problems like Deep Deterministic Policy Gradient (DDPG) , Asynchronous Advantage Actor Critic (A3C) BID11 use actor-critic architectures, where an action function is learned by mapping states to actions. DDPG works well on many tasks, but it does not model the uncertainty in actions as it produces a point estimate of the action distribution over states. The actor is forced to deterministically choose an action for every state. A3C and other stochastic policy gradient algorithms output distribution parameters (e.g. Gaussian distributions) instead of point estimate, which can be sampled for action values.As a simple example where this is sub-optimal, consider the inverted pendulum task, where a pendulum is attached to a cart and the agent needs to control the one dimensional movement of the cart to balance the pendulum upside down. A deterministic agent chooses a single action for every state. This breaks the inherent symmetry of the task. When the cart in not moving and the pendulum is hanging down, two actions are equally promising: either moving left or right. The distribution parameter estimation (e.g. A3C) might work better in this case as there are only two good options, but in cases when there are more than two good actions to select, this will not be optimal. In our approach we allow the agent to suggest multiple actions, which enables it to resolve cases like this easily.Further, we observe that a deterministic behavior of DDPG can lead to sub-optimal convergence during training. The main limitation is that, especially in the beginning of the learning procedure, the actor favors actions that lead to a good immediate reward but might end up being far from the globally optimal choice. This work is based on the intuition that if the actor is allowed to suggest, at each time step, multiple actions rather than a single one, this can render the resulting policy non-deterministic, leading to a better exploration of the entire solution space as well as a final solution of potentially higher quality. This can also eliminate the external exploration mechanisms required during training e.g. OrnsteinUhlenbeck process noise BID20 , parameter noise or differential entropy of normal distribution.Here, we introduce an algorithm, which we refer to as Multiple Action Policy Gradients (MAPG), that models a stochastic policy with several point estimates and allows to predict a pre-defined number M of actions at each time step, extending any policy gradient algorithm with little overhead. We will demonstrate the working of this algorithm by adapting DDPG Lillicrap et al. (2016) to use MAPG.Another benefit of the proposed method is that the variance of the predicted actions can give additional insights into the decision process during runtime. A low variance usually implies that the model only sees one way to act in a certain situation. A wider or even multi-modal distribution suggests that there exist several possibilities given the current state.We evaluate the proposed method on six continuous control problems of the OpenAI Gym BID0 as well as a deep driving scenario using the TORCS car simulator BID22 . For a fair evaluation we directly compare DDPG to our MAPG without changing hyper-parameters or modifying the training scheme. In all experiments, we show an improved performance using MAPG over DDPG. To verify if MAPG helps in better exploration during training, we also analyze MAPG under no external exploration policy. In this paper, we have proposed MAPG, a technique that leverages multiple action prediction to learn better policies in continuous control problems. The proposed method enables a better exploration of the state space and shows improved performance over DDPG. As indicated by exploration experiments, it can also be a used as a standalone exploration technique, although more work needs to be done in this direction. Last but not least, we conclude with interesting insights gained from the action variance. There are several interesting directions which we would like to investigate in the future. The number of actions M is a hyper-parameter in our model that needs to be selected and seems to be task specific. In general, the idea of predicting multiple action proposals can be extended to other on-or off-policy algorithms, such as NAF BID1 or TRPO. Evaluating MA-NAF and MA-TRPO will enable studying the generality of the proposed approach.
We introduce a novel reinforcement learning algorithm, that predicts multiple actions and samples from them. This work introduces a uniform mixture of deterministic policies, and find that this parametrization of stochastic policies outperforms DDPG on several OpenAI gym benchmarks. The authors investigate a method for improving the performance of networks trained with DDPG, and show improved performance on a large number of standard continuous control environment.