Fill-and-Spill: Deep Reinforcement Learning Policy Gradient Methods for
  Reservoir Operation Decision and Control
Sadegh Sadeghi Tabas Vidya Samadi
Changes in demand, various hydrological inputs, and environmental stressors
are among the issues that water managers and policymakers face on a regular
basis. These concerns have sparked interest in applying different techniques to
determine reservoir operation policy decisions. As the resolution of the
analysis increases, it becomes more difficult to effectively represent a
real-world system using traditional methods such as Dynamic Programming (DP)
and Stochastic Dynamic Programming (SDP) for determining the best reservoir
operation policy. One of the challenges is the "curse of dimensionality," which
means the number of samples needed to estimate an arbitrary function with a
given level of accuracy grows exponentially with respect to the number of input
variables (i.e., dimensionality) of the function. Deep Reinforcement Learning
(DRL) is an intelligent approach to overcome the curses of stochastic
optimization problems for reservoir operation policy decisions. To our
knowledge, this study is the first attempt that examine various novel DRL
continuous-action policy gradient methods (PGMs), including Deep Deterministic
Policy Gradients (DDPG), Twin Delayed DDPG (TD3), and two different versions of
Soft Actor-Critic (SAC18 and SAC19) for optimizing reservoir operation policy.
In this study, multiple DRL techniques were implemented in order to find the
optimal operation policy of Folsom Reservoir in California, USA. The reservoir
system supplies agricultural, municipal, hydropower, and environmental flow
demands and flood control operations to the City of Sacramento. Analysis
suggests that the TD3 and SAC are robust to meet the Folsom Reservoir's demands
and optimize reservoir operation policies.