Multi-Agent Reinforcement Learning for Offloading Cellular
  Communications with Cooperating UAVs
Abhishek Mondal Deepak Mishra Ganesh Prasad George C. Alexandropoulos Azzam Alnahari Riku Jantti
Effective solutions for intelligent data collection in terrestrial cellular
networks are crucial, especially in the context of Internet of Things
applications. The limited spectrum and coverage area of terrestrial base
stations pose challenges in meeting the escalating data rate demands of network
users. Unmanned aerial vehicles, known for their high agility, mobility, and
flexibility, present an alternative means to offload data traffic from
terrestrial BSs, serving as additional access points. This paper introduces a
novel approach to efficiently maximize the utilization of multiple UAVs for
data traffic offloading from terrestrial BSs. Specifically, the focus is on
maximizing user association with UAVs by jointly optimizing UAV trajectories
and users association indicators under quality of service constraints. Since,
the formulated UAVs control problem is nonconvex and combinatorial, this study
leverages the multi agent reinforcement learning framework. In this framework,
each UAV acts as an independent agent, aiming to maintain inter UAV cooperative
behavior. The proposed approach utilizes the finite state Markov decision
process to account for UAVs velocity constraints and the relationship between
their trajectories and state space. A low complexity distributed state action
reward state action algorithm is presented to determine UAVs optimal sequential
decision making policies over training episodes. The extensive simulation
results validate the proposed analysis and offer valuable insights into the
optimal UAV trajectories. The derived trajectories demonstrate superior average
UAV association performance compared to benchmark techniques such as Q learning
and particle swarm optimization.