Improved Long Short-Term Memory-based Wastewater Treatment Simulators
  for Deep Reinforcement Learning
Esmaeel Mohammadi Daniel Ortiz-Arroyo Mikkel Stokholm-Bjerregaard Aviaja Anna Hansen Petar Durdevic
Even though Deep Reinforcement Learning (DRL) showed outstanding results in
the fields of Robotics and Games, it is still challenging to implement it in
the optimization of industrial processes like wastewater treatment. One of the
challenges is the lack of a simulation environment that will represent the
actual plant as accurately as possible to train DRL policies. Stochasticity and
non-linearity of wastewater treatment data lead to unstable and incorrect
predictions of models over long time horizons. One possible reason for the
models' incorrect simulation behavior can be related to the issue of
compounding error, which is the accumulation of errors throughout the
simulation. The compounding error occurs because the model utilizes its
predictions as inputs at each time step. The error between the actual data and
the prediction accumulates as the simulation continues. We implemented two
methods to improve the trained models for wastewater treatment data, which
resulted in more accurate simulators: 1- Using the model's prediction data as
input in the training step as a tool of correction, and 2- Change in the loss
function to consider the long-term predicted shape (dynamics). The experimental
results showed that implementing these methods can improve the behavior of
simulators in terms of Dynamic Time Warping throughout a year up to 98%
compared to the base model. These improvements demonstrate significant promise
in creating simulators for biological processes that do not need pre-existing
knowledge of the process but instead depend exclusively on time series data
obtained from the system.