Enhancing Continuous Time Series Modelling with a Latent ODE-LSTM
  Approach
C. Coelho M. Fernanda P. Costa L. L. Ferrás
Due to their dynamic properties such as irregular sampling rate and
high-frequency sampling, Continuous Time Series (CTS) are found in many
applications. Since CTS with irregular sampling rate are difficult to model
with standard Recurrent Neural Networks (RNNs), RNNs have been generalised to
have continuous-time hidden dynamics defined by a Neural Ordinary Differential
Equation (Neural ODE), leading to the ODE-RNN model. Another approach that
provides a better modelling is that of the Latent ODE model, which constructs a
continuous-time model where a latent state is defined at all times. The Latent
ODE model uses a standard RNN as the encoder and a Neural ODE as the decoder.
However, since the RNN encoder leads to difficulties with missing data and
ill-defined latent variables, a Latent ODE-RNN model has recently been proposed
that uses a ODE-RNN model as the encoder instead. Both the Latent ODE and
Latent ODE-RNN models are difficult to train due to the vanishing and exploding
gradients problem. To overcome this problem, the main contribution of this
paper is to propose and illustrate a new model based on a new Latent ODE using
an ODE-LSTM (Long Short-Term Memory) network as an encoder -- the Latent
ODE-LSTM model. To limit the growth of the gradients the Norm Gradient Clipping
strategy was embedded on the Latent ODE-LSTM model. The performance evaluation
of the new Latent ODE-LSTM (with and without Norm Gradient Clipping) for
modelling CTS with regular and irregular sampling rates is then demonstrated.
Numerical experiments show that the new Latent ODE-LSTM performs better than
Latent ODE-RNNs and can avoid the vanishing and exploding gradients during
training.