Hybrid State Space-based Learning for Sequential Data Prediction with
  Joint Optimization
Mustafa E. Aydın Arda Fazla Suleyman S. Kozat
We investigate nonlinear prediction/regression in an online setting and
introduce a hybrid model that effectively mitigates, via a joint mechanism
through a state space formulation, the need for domain-specific feature
engineering issues of conventional nonlinear prediction models and achieves an
efficient mix of nonlinear and linear components. In particular, we use
recursive structures to extract features from raw sequential sequences and a
traditional linear time series model to deal with the intricacies of the
sequential data, e.g., seasonality, trends. The state-of-the-art ensemble or
hybrid models typically train the base models in a disjoint manner, which is
not only time consuming but also sub-optimal due to the separation of modeling
or independent training. In contrast, as the first time in the literature, we
jointly optimize an enhanced recurrent neural network (LSTM) for automatic
feature extraction from raw data and an ARMA-family time series model (SARIMAX)
for effectively addressing peculiarities associated with time series data. We
achieve this by introducing novel state space representations for the base
models, which are then combined to provide a full state space representation of
the hybrid or the ensemble. Hence, we are able to jointly optimize both models
in a single pass via particle filtering, for which we also provide the update
equations. The introduced architecture is generic so that one can use other
recurrent architectures, e.g., GRUs, traditional time series-specific models,
e.g., ETS or other optimization methods, e.g., EKF, UKF. Due to such novel
combination and joint optimization, we demonstrate significant improvements in
widely publicized real life competition datasets. We also openly share our code
for further research and replicability of our results.