A Bayesian Framework of Deep Reinforcement Learning for Joint O-RAN/MEC
  Orchestration
Fahri Wisnu Murti Samad Ali Matti Latva-aho
Multi-access Edge Computing (MEC) can be implemented together with Open Radio
Access Network (O-RAN) over commodity platforms to offer low-cost deployment
and bring the services closer to end-users. In this paper, a joint O-RAN/MEC
orchestration using a Bayesian deep reinforcement learning (RL)-based framework
is proposed that jointly controls the O-RAN functional splits, the allocated
resources and hosting locations of the O-RAN/MEC services across
geo-distributed platforms, and the routing for each O-RAN/MEC data flow. The
goal is to minimize the long-term overall network operation cost and maximize
the MEC performance criterion while adapting possibly time-varying O-RAN/MEC
demands and resource availability. This orchestration problem is formulated as
Markov decision process (MDP). However, the system consists of multiple BSs
that share the same resources and serve heterogeneous demands, where their
parameters have non-trivial relations. Consequently, finding the exact model of
the underlying system is impractical, and the formulated MDP renders in a large
state space with multi-dimensional discrete action. To address such modeling
and dimensionality issues, a novel model-free RL agent is proposed for our
solution framework. The agent is built from Double Deep Q-network (DDQN) that
tackles the large state space and is then incorporated with action branching,
an action decomposition method that effectively addresses the multi-dimensional
discrete action with linear increase complexity. Further, an efficient
exploration-exploitation strategy under a Bayesian framework using Thomson
sampling is proposed to improve the learning performance and expedite its
convergence. Trace-driven simulations are performed using an O-RAN-compliant
model. The results show that our approach is data-efficient (i.e., converges
faster) and increases the returned reward by 32\% than its non-Bayesian
version.