Improving the Performance of Echo State Networks Through Feedback
Peter J. Ehlers Hendra I. Nurdin Daniel Soh
Reservoir computing, using nonlinear dynamical systems, offers a
cost-effective alternative to neural networks for complex tasks involving
processing of sequential data, time series modeling, and system identification.
Echo state networks (ESNs), a type of reservoir computer, mirror neural
networks but simplify training. They apply fixed, random linear transformations
to the internal state, followed by nonlinear changes. This process, guided by
input signals and linear regression, adapts the system to match target
characteristics, reducing computational demands. A potential drawback of ESNs
is that the fixed reservoir may not offer the complexity needed for specific
problems. While directly altering (training) the internal ESN would reintroduce
the computational burden, an indirect modification can be achieved by
redirecting some output as input. This feedback can influence the internal
reservoir state, yielding ESNs with enhanced complexity suitable for broader
challenges. In this paper, we demonstrate that by feeding some component of the
reservoir state back into the network through the input, we can drastically
improve upon the performance of a given ESN. We rigorously prove that, for any
given ESN, feedback will almost always improve the accuracy of the output. For
a set of three tasks, each representing different problem classes, we find that
with feedback the average error measures are reduced by $30\%-60\%$.
Remarkably, feedback provides at least an equivalent performance boost to
doubling the initial number of computational nodes, a computationally expensive
and technologically challenging alternative. These results demonstrate the
broad applicability and substantial usefulness of this feedback scheme.