Multi-Timescale Control and Communications with Deep Reinforcement
  Learning -- Part II: Control-Aware Radio Resource Allocation
Lei Lei Tong Liu Kan Zheng Xuemin Shen
In Part I of this two-part paper (Multi-Timescale Control and Communications
with Deep Reinforcement Learning -- Part I: Communication-Aware Vehicle
Control), we decomposed the multi-timescale control and communications (MTCC)
problem in Cellular Vehicle-to-Everything (C-V2X) system into a
communication-aware Deep Reinforcement Learning (DRL)-based platoon control
(PC) sub-problem and a control-aware DRL-based radio resource allocation (RRA)
sub-problem. We focused on the PC sub-problem and proposed the MTCC-PC
algorithm to learn an optimal PC policy given an RRA policy. In this paper
(Part II), we first focus on the RRA sub-problem in MTCC assuming a PC policy
is given, and propose the MTCC-RRA algorithm to learn the RRA policy.
Specifically, we incorporate the PC advantage function in the RRA reward
function, which quantifies the amount of PC performance degradation caused by
observation delay. Moreover, we augment the state space of RRA with PC action
history for a more well-informed RRA policy. In addition, we utilize reward
shaping and reward backpropagation prioritized experience replay (RBPER)
techniques to efficiently tackle the multi-agent and sparse reward problems,
respectively. Finally, a sample- and computational-efficient training approach
is proposed to jointly learn the PC and RRA policies in an iterative process.
In order to verify the effectiveness of the proposed MTCC algorithm, we
performed experiments using real driving data for the leading vehicle, where
the performance of MTCC is compared with those of the baseline DRL algorithms.