DHRL-FNMR: An Intelligent Multicast Routing Approach Based on Deep
  Hierarchical Reinforcement Learning in SDN
Miao Ye Chenwei Zhao Xingsi Xue Jinqiang Li Hongwen Hu Yejin Yang Qiuxiang Jiang
The optimal multicast tree problem in the Software-Defined Networking (SDN)
multicast routing is an NP-hard combinatorial optimization problem. Although
existing SDN intelligent solution methods, which are based on deep
reinforcement learning, can dynamically adapt to complex network link state
changes, these methods are plagued by problems such as redundant branches,
large action space, and slow agent convergence. In this paper, an SDN
intelligent multicast routing algorithm based on deep hierarchical
reinforcement learning is proposed to circumvent the aforementioned problems.
First, the multicast tree construction problem is decomposed into two
sub-problems: the fork node selection problem and the construction of the
optimal path from the fork node to the destination node. Second, based on the
information characteristics of SDN global network perception, the multicast
tree state matrix, link bandwidth matrix, link delay matrix, link packet loss
rate matrix, and sub-goal matrix are designed as the state space of intrinsic
and meta controllers. Then, in order to mitigate the excessive action space,
our approach constructs different action spaces at the upper and lower levels.
The meta-controller generates an action space using network nodes to select the
fork node, and the intrinsic controller uses the adjacent edges of the current
node as its action space, thus implementing four different action selection
strategies in the construction of the multicast tree. To facilitate the
intelligent agent in constructing the optimal multicast tree with greater
speed, we developed alternative reward strategies that distinguish between
single-step node actions and multi-step actions towards multiple destination
nodes.