Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning
  for Digital Twins
Eslam Eldeeb Houssem Sifaou Osvaldo Simeone Mohammad Shehab Hirley Alves
Digital twin (DT) platforms are increasingly regarded as a promising
technology for controlling, optimizing, and monitoring complex engineering
systems such as next-generation wireless networks. An important challenge in
adopting DT solutions is their reliance on data collected offline, lacking
direct access to the physical environment. This limitation is particularly
severe in multi-agent systems, for which conventional multi-agent reinforcement
(MARL) requires online interactions with the environment. A direct application
of online MARL schemes to an offline setting would generally fail due to the
epistemic uncertainty entailed by the limited availability of data. In this
work, we propose an offline MARL scheme for DT-based wireless networks that
integrates distributional RL and conservative Q-learning to address the
environment's inherent aleatoric uncertainty and the epistemic uncertainty
arising from limited data. To further exploit the offline data, we adapt the
proposed scheme to the centralized training decentralized execution framework,
allowing joint training of the agents' policies. The proposed MARL scheme,
referred to as multi-agent conservative quantile regression (MA-CQR) addresses
general risk-sensitive design criteria and is applied to the trajectory
planning problem in drone networks, showcasing its advantages.