Multi-agent Reinforcement Learning for Energy Saving in Multi-Cell
  Massive MIMO Systems
Tianzhang Cai Qichen Wang Shuai Zhang Özlem Tuğfe Demir Cicek Cavdar
We develop a multi-agent reinforcement learning (MARL) algorithm to minimize
the total energy consumption of multiple massive MIMO (multiple-input
multiple-output) base stations (BSs) in a multi-cell network while preserving
the overall quality-of-service (QoS) by making decisions on the multi-level
advanced sleep modes (ASMs) and antenna switching of these BSs. The problem is
modeled as a decentralized partially observable Markov decision process
(DEC-POMDP) to enable collaboration between individual BSs, which is necessary
to tackle inter-cell interference. A multi-agent proximal policy optimization
(MAPPO) algorithm is designed to learn a collaborative BS control policy. To
enhance its scalability, a modified version called MAPPO-neighbor policy is
further proposed. Simulation results demonstrate that the trained MAPPO agent
achieves better performance compared to baseline policies. Specifically,
compared to the auto sleep mode 1 (symbol-level sleeping) algorithm, the
MAPPO-neighbor policy reduces power consumption by approximately 8.7% during
low-traffic hours and improves energy efficiency by approximately 19% during
high-traffic hours, respectively.