Digital Twin-Enhanced Deep Reinforcement Learning for Resource
  Management in Networks Slicing
Zhengming Zhang Yongming Huang Cheng Zhang Qingbi Zheng Luxi Yang Xiaohu You
Network slicing-based communication systems can dynamically and efficiently
allocate resources for diversified services. However, due to the limitation of
the network interface on channel access and the complexity of the resource
allocation, it is challenging to achieve an acceptable solution in the
practical system without precise prior knowledge of the dynamics probability
model of the service requests. Existing work attempts to solve this problem
using deep reinforcement learning (DRL), however, such methods usually require
a lot of interaction with the real environment in order to achieve good
results. In this paper, a framework consisting of a digital twin and
reinforcement learning agents is present to handle the issue. Specifically, we
propose to use the historical data and the neural networks to build a digital
twin model to simulate the state variation law of the real environment. Then,
we use the data generated by the network slicing environment to calibrate the
digital twin so that it is in sync with the real environment. Finally, DRL for
slice optimization optimizes its own performance in this virtual
pre-verification environment. We conducted an exhaustive verification of the
proposed digital twin framework to confirm its scalability. Specifically, we
propose to use loss landscapes to visualize the generalization of DRL
solutions. We explore a distillation-based optimization scheme for lightweight
slicing strategies. In addition, we also extend the framework to offline
reinforcement learning, where solutions can be used to obtain intelligent
decisions based solely on historical data. Numerical simulation experiments
show that the proposed digital twin can significantly improve the performance
of the slice optimization strategy.