Decentralized Multi-agent Reinforcement Learning based State-of-Charge
  Balancing Strategy for Distributed Energy Storage System
Zheng Xiong Biao Luo Bing-Chuan Wang Xiaodong Xu Xiaodong Liu Tingwen Huang
This paper develops a Decentralized Multi-Agent Reinforcement Learning
(Dec-MARL) method to solve the SoC balancing problem in the distributed energy
storage system (DESS). First, the SoC balancing problem is formulated into a
finite Markov decision process with action constraints derived from demand
balance, which can be solved by Dec-MARL. Specifically, the first-order average
consensus algorithm is utilized to expand the observations of the DESS state in
a fully-decentralized way, and the initial actions (i.e., output power) are
decided by the agents (i.e., energy storage units) according to these
observations. In order to get the final actions in the allowable range, a
counterfactual demand balance algorithm is proposed to balance the total demand
and the initial actions. Next, the agents execute the final actions and get
local rewards from the environment, and the DESS steps into the next state.
Finally, through the first-order average consensus algorithm, the agents get
the average reward and the expended observation of the next state for later
training. By the above procedure, Dec-MARL reveals outstanding performance in a
fully-decentralized system without any expert experience or constructing any
complicated model. Besides, it is flexible and can be extended to other
decentralized multi-agent systems straightforwardly. Extensive simulations have
validated the effectiveness and efficiency of Dec-MARL.