Multi-Timescale Control and Communications with Deep Reinforcement
  Learning -- Part I: Communication-Aware Vehicle Control
Tong Liu Lei Lei Kan Zheng Xuemin Shen
An intelligent decision-making system enabled by Vehicle-to-Everything (V2X)
communications is essential to achieve safe and efficient autonomous driving
(AD), where two types of decisions have to be made at different timescales,
i.e., vehicle control and radio resource allocation (RRA) decisions. The
interplay between RRA and vehicle control necessitates their collaborative
design. In this two-part paper (Part I and Part II), taking platoon control
(PC) as an example use case, we propose a joint optimization framework of
multi-timescale control and communications (MTCC) based on Deep Reinforcement
Learning (DRL). In this paper (Part I), we first decompose the problem into a
communication-aware DRL-based PC sub-problem and a control-aware DRL-based RRA
sub-problem. Then, we focus on the PC sub-problem assuming an RRA policy is
given, and propose the MTCC-PC algorithm to learn an efficient PC policy. To
improve the PC performance under random observation delay, the PC state space
is augmented with the observation delay and PC action history. Moreover, the
reward function with respect to the augmented state is defined to construct an
augmented state Markov Decision Process (MDP). It is proved that the optimal
policy for the augmented state MDP is optimal for the original PC problem with
observation delay. Different from most existing works on communication-aware
control, the MTCC-PC algorithm is trained in a delayed environment generated by
the fine-grained embedded simulation of C-V2X communications rather than by a
simple stochastic delay model. Finally, experiments are performed to compare
the performance of MTCC-PC with those of the baseline DRL algorithms.