Effective Reinforcement Learning Based on Structural Information
  Principles
Xianghua Zeng Hao Peng Dingli Su Angsheng Li
Although Reinforcement Learning (RL) algorithms acquire sequential behavioral
patterns through interactions with the environment, their effectiveness in
noisy and high-dimensional scenarios typically relies on specific structural
priors. In this paper, we propose a novel and general Structural Information
principles-based framework for effective Decision-Making, namely SIDM,
approached from an information-theoretic perspective. This paper presents a
specific unsupervised partitioning method that forms vertex communities in the
state and action spaces based on their feature similarities. An aggregation
function, which utilizes structural entropy as the vertex weight, is devised
within each community to obtain its embedding, thereby facilitating
hierarchical state and action abstractions. By extracting abstract elements
from historical trajectories, a directed, weighted, homogeneous transition
graph is constructed. The minimization of this graph's high-dimensional entropy
leads to the generation of an optimal encoding tree. An innovative two-layer
skill-based learning mechanism is introduced to compute the common path entropy
of each state transition as its identified probability, thereby obviating the
requirement for expert knowledge. Moreover, SIDM can be flexibly incorporated
into various single-agent and multi-agent RL algorithms, enhancing their
performance. Finally, extensive evaluations on challenging benchmarks
demonstrate that, compared with SOTA baselines, our framework significantly and
consistently improves the policy's quality, stability, and efficiency up to
32.70%, 88.26%, and 64.86%, respectively.