Message Propagation Through Time: An Algorithm for Sequence Dependency
  Retention in Time Series Modeling
Shaoming Xu Ankush Khandelwal Arvind Renganathan Vipin Kumar
Time series modeling, a crucial area in science, often encounters challenges
when training Machine Learning (ML) models like Recurrent Neural Networks
(RNNs) using the conventional mini-batch training strategy that assumes
independent and identically distributed (IID) samples and initializes RNNs with
zero hidden states. The IID assumption ignores temporal dependencies among
samples, resulting in poor performance. This paper proposes the Message
Propagation Through Time (MPTT) algorithm to effectively incorporate long
temporal dependencies while preserving faster training times relative to the
stateful solutions. MPTT utilizes two memory modules to asynchronously manage
initial hidden states for RNNs, fostering seamless information exchange between
samples and allowing diverse mini-batches throughout epochs. MPTT further
implements three policies to filter outdated and preserve essential information
in the hidden states to generate informative initial hidden states for RNNs,
facilitating robust training. Experimental results demonstrate that MPTT
outperforms seven strategies on four climate datasets with varying levels of
temporal dependencies.