Towards Safe Load Balancing based on Control Barrier Functions and Deep
  Reinforcement Learning
Lam Dinh Pham Tran Anh Quang Jérémie Leguay
Deep Reinforcement Learning (DRL) algorithms have recently made significant
strides in improving network performance. Nonetheless, their practical use is
still limited in the absence of safe exploration and safe decision-making. In
the context of commercial solutions, reliable and safe-to-operate systems are
of paramount importance. Taking this problem into account, we propose a safe
learning-based load balancing algorithm for Software Defined-Wide Area Network
(SD-WAN), which is empowered by Deep Reinforcement Learning (DRL) combined with
a Control Barrier Function (CBF). It safely projects unsafe actions into
feasible ones during both training and testing, and it guides learning towards
safe policies. We successfully implemented the solution on GPU to accelerate
training by approximately 110x times and achieve model updates for on-policy
methods within a few seconds, making the solution practical. We show that our
approach delivers near-optimal Quality-of-Service (QoS performance in terms of
end-to-end delay while respecting safety requirements related to link capacity
constraints. We also demonstrated that on-policy learning based on Proximal
Policy Optimization (PPO) performs better than off-policy learning with Deep
Deterministic Policy Gradient (DDPG) when both are combined with a CBF for safe
load balancing.