A Constraint Enforcement Deep Reinforcement Learning Framework for
  Optimal Energy Storage Systems Dispatch
Shengren Hou Edgar Mauricio Salazar Duque Peter Palensky Pedro P. Vergara
The optimal dispatch of energy storage systems (ESSs) presents formidable
challenges due to the uncertainty introduced by fluctuations in dynamic prices,
demand consumption, and renewable-based energy generation. By exploiting the
generalization capabilities of deep neural networks (DNNs), deep reinforcement
learning (DRL) algorithms can learn good-quality control models that adaptively
respond to distribution networks' stochastic nature. However, current DRL
algorithms lack the capabilities to enforce operational constraints strictly,
often even providing unfeasible control actions. To address this issue, we
propose a DRL framework that effectively handles continuous action spaces while
strictly enforcing the environments and action space operational constraints
during online operation. Firstly, the proposed framework trains an action-value
function modeled using DNNs. Subsequently, this action-value function is
formulated as a mixed-integer programming (MIP) formulation enabling the
consideration of the environment's operational constraints. Comprehensive
numerical simulations show the superior performance of the proposed MIP-DRL
framework, effectively enforcing all constraints while delivering high-quality
dispatch decisions when compared with state-of-the-art DRL algorithms and the
optimal solution obtained with a perfect forecast of the stochastic variables.