Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided. However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization. This limits the applicability of RL in the real world since both reinforcement learning and domain-specific knowledge are required. It is therefore of great practical importance to develop algorithms which can learn from a binary signal indicating successful task completion or other unshaped, sparse reward signals. We propose a novel method called competitive experience replay, which efficiently supplements a sparse reward by placing learning in the context of an exploration competition between a pair of agents. Our method complements the recently proposed hindsight experience replay (HER) by inducing an automatic exploratory curriculum. We evaluate our approach on the tasks of reaching various goal locations in an ant maze and manipulating objects with a robotic arm. Each task provides only binary rewards indicating whether or not the goal is achieved. Our method asymmetrically augments these sparse rewards for a pair of agents each learning the same task, creating a competitive game designed to drive exploration. Extensive experiments demonstrate that this method leads to faster converge and improved task performance. Recent progress in deep reinforcement learning has achieved very impressive results in domains ranging from playing games BID28 BID39 BID31 , to high dimensional continuous control , and robotics BID21 BID2 .Despite these successes, in robotics control and many other areas, deep reinforcement learning still suffers from the need to engineer a proper reward function to guide policy optimization (see e.g. BID36 BID29 . In robotic control as stacking bricks, reward function need to be sharped to very complex which consists of multiple terms BID36 . It is extremely hard and not applicable to engineer such reward function for each task in real world since both reinforcement learning expertise and domain-specific knowledge are required. Learning to perform well in environments with sparse rewards remains a major challenge. Therefore, it is of great practical importance to develop algorithms which can learn from binary signal indicating successful task completion or other unshaped reward signal.In environments where dense reward function is not available, only a small fraction of the agents' experiences will be useful to compute gradient to optimize policy, leading to substantial high sample complexity. Providing agents with useful signals to pursue in sparse reward environments becomes crucial in these scenarios.In the domain of goal-directed RL, the recently proposed hindsight experience replay (HER) BID0 addresses the challenge of learning from sparse rewards by re-labelling visited states as goal states during training. However, this technique continues to suffer from sample inefficiency, ostensibly due to difficulties related to exploration. In this work, we address these limitations by introducing a method called Competitive Experience Replay (CER). This technique attempts to emphasize exploration by introducing a competition between two agents attempting to learn the same task. Intuitively, agent A (the agent ultimately used for evaluation) receives a penalty for visiting states that the competitor agent (B) also visits; and B is rewarded for visiting states found by A. Our approach maintains the reward from the original task such that exploration is biased towards the behaviors best suited to accomplishing the task goals. We show that this competition between agents can automatically generate a curriculum of exploration and shape otherwise sparse reward. We jointly train both agents' policies by adopting methods from multi-agent RL. In addition, we propose two versions of CER, independent CER, and interact CER, which differ in the state initialization of agent B: whether it is sampled from the initial state distribution or sampled from off-policy samples of agent A, respectively.Whereas HER re-labels samples based on an agent's individual rollout, our method re-labels samples based on intra-agent behavior; as such, the two methods do not interfere with each other algorithmically and are easily combined during training. We evaluate our method both with and without HER on a variety of reinforcement learning tasks, including navigating an ant agent to reach a goal position and manipulating objects with a robotic arm. For each such task the default reward is sparse, corresponding to a binary indicator of goal completion. Ablation studies show that our method is important for achieving a high success rate and often demonstrates faster convergence. Interestingly, we find that CER and HER are complementary methods and employ both to reach peak efficiency and performance. Furthermore, we observe that, when combined with HER, CER outperforms curiosity-driven exploration. We introduce Competitive Experience Replay, a new and general method for encouraging exploration through implicit curriculum learning in sparse reward settings. We demonstrate an empirical advantage of our technique when combined with existing methods in several challenging RL tasks. In future work, we aim to investigate richer ways to re-label rewards based on intra-agent samples to further harness multi-agent competition, it's interesting to investigate counterfactual inference to promote efficient re-label off-policy samples. We hope that this will facilitate the application of our method to more open-end environments with even more challenging task structures. In addition, future work will explore integrating our method into approaches more closely related to model-based learning, where adequate exposure to the dynamics of the environment is often crucial. As BID0 and BID1 observe (and we also observe), performance depends on the batch size. We leverage this observation to tune the relative strengths of Agents A and B by separately manipulating the batch sizes used for updating each.For simplicity, we control the batch size by changing the number of MPI workers devoted to a particular update. Each MPI worker computes the gradients for a batch size of 256; averaging the gradients from each worker results in an effective batch size of N * 256. For our single-agent baselines, we choose N = 30 workers, and, when using CER, a default of N = 15 for each A and B In the following, AxBy denotes, for agent A, N = x and, for agent B, N = y. These results suggest that, while a sufficiently large batch size is important for achieving the best performance, the optimal configuration occurs when the batch sizes used for the two agents are balanced. Interestingly, we observe that batch size imbalance adversely effects both agents trained during CER.
a novel method to learn with sparse reward using adversarial reward re-labeling Proposes to use a competitive multi-agent setting for encouraging exploration and shows that CER + HER > HER ~ CER Propose a new method for learning from sparse rewards in model-free reinforcement learning settings and densifying reward To address the sparse reward problems and encourage exploration in RL algorithms, the authors propose a relabeling strategy called Competitive Experience Reply (CER).