Causal Coordinated Concurrent Reinforcement Learning
Tim Tse Isaac Chan Zhitang Chen
In this work, we propose a novel algorithmic framework for data sharing and
coordinated exploration for the purpose of learning more data-efficient and
better performing policies under a concurrent reinforcement learning (CRL)
setting. In contrast to other work which make the assumption that all agents
act under identical environments, we relax this restriction and instead
consider the formulation where each agent acts within an environment which
shares a global structure but also exhibits individual variations. Our
algorithm leverages a causal inference algorithm in the form of Additive Noise
Model - Mixture Model (ANM-MM) in extracting model parameters governing
individual differentials via independence enforcement. We propose a new data
sharing scheme based on a similarity measure of the extracted model parameters
and demonstrate superior learning speeds on a set of autoregressive, pendulum
and cart-pole swing-up tasks and finally, we show the effectiveness of diverse
action selection between common agents under a sparse reward setting. To the
best of our knowledge, this is the first work in considering non-identical
environments in CRL and one of the few works which seek to integrate causal
inference with reinforcement learning (RL).