Communication-Efficient Orchestrations for URLLC Service via
  Hierarchical Reinforcement Learning
Wei Shi Milad Ganjalizadeh Hossein Shokri Ghadikolaei Marina Petrova
Ultra-reliable low latency communications (URLLC) service is envisioned to
enable use cases with strict reliability and latency requirements in 5G. One
approach for enabling URLLC services is to leverage Reinforcement Learning (RL)
to efficiently allocate wireless resources. However, with conventional RL
methods, the decision variables (though being deployed at various network
layers) are typically optimized in the same control loop, leading to
significant practical limitations on the control loop's delay as well as
excessive signaling and energy consumption. In this paper, we propose a
multi-agent Hierarchical RL (HRL) framework that enables the implementation of
multi-level policies with different control loop timescales. Agents with faster
control loops are deployed closer to the base station, while the ones with
slower control loops are at the edge or closer to the core network providing
high-level guidelines for low-level actions. On a use case from the prior art,
with our HRL framework, we optimized the maximum number of retransmissions and
transmission power of industrial devices. Our extensive simulation results on
the factory automation scenario show that the HRL framework achieves better
performance as the baseline single-agent RL method, with significantly less
overhead of signal transmissions and delay compared to the one-agent RL
methods.