Intervention-Assisted Policy Gradient Methods for Online Stochastic
  Queuing Network Optimization: Technical Report
Jerrod Wigmore Brooke Shrader Eytan Modiano
Deep Reinforcement Learning (DRL) offers a powerful approach to training
neural network control policies for stochastic queuing networks (SQN). However,
traditional DRL methods rely on offline simulations or static datasets,
limiting their real-world application in SQN control. This work proposes Online
Deep Reinforcement Learning-based Controls (ODRLC) as an alternative, where an
intelligent agent interacts directly with a real environment and learns an
optimal control policy from these online interactions. SQNs present a challenge
for ODRLC due to the unbounded nature of the queues within the network
resulting in an unbounded state-space. An unbounded state-space is particularly
challenging for neural network policies as neural networks are notoriously poor
at extrapolating to unseen states. To address this challenge, we propose an
intervention-assisted framework that leverages strategic interventions from
known stable policies to ensure the queue sizes remain bounded. This framework
combines the learning power of neural networks with the guaranteed stability of
classical control policies for SQNs. We introduce a method to design these
intervention-assisted policies to ensure strong stability of the network.
Furthermore, we extend foundational DRL theorems for intervention-assisted
policies and develop two practical algorithms specifically for ODRLC of SQNs.
Finally, we demonstrate through experiments that our proposed algorithms
outperform both classical control approaches and prior ODRLC algorithms.