A Scalable and Parallelizable Digital Twin Framework for Sustainable
  Sim2Real Transition of Multi-Agent Reinforcement Learning Systems
Chinmay Vilas Samak Tanmay Vilas Samak Venkat Krovi
This work presents a sustainable multi-agent deep reinforcement learning
framework capable of selectively scaling parallelized training workloads
on-demand, and transferring the trained policies from simulation to reality
using minimal hardware resources. We introduce AutoDRIVE Ecosystem as an
enabling digital twin framework to train, deploy, and transfer cooperative as
well as competitive multi-agent reinforcement learning policies from simulation
to reality. Particularly, we first investigate an intersection traversal
problem of 4 cooperative vehicles (Nigel) that share limited state information
in single as well as multi-agent learning settings using a common policy
approach. We then investigate an adversarial autonomous racing problem of 2
vehicles (F1TENTH) using an individual policy approach. In either set of
experiments, a decentralized learning architecture was adopted, which allowed
robust training and testing of the policies in stochastic environments. The
agents were provided with realistically sparse observation spaces, and were
restricted to sample control actions that implicitly satisfied the imposed
kinodynamic and safety constraints. The experimental results for both problem
statements are reported in terms of quantitative metrics and qualitative
remarks for training as well as deployment phases. We also discuss agent and
environment parallelization techniques adopted to efficiently accelerate MARL
training, while analyzing their computational performance. Finally, we
demonstrate a resource-aware transition of the trained policies from simulation
to reality using the proposed digital twin framework.