Rough Transformers for Continuous and Efficient Time-Series Modelling
Fernando Moreno-Pino Álvaro Arroyo Harrison Waldon Xiaowen Dong Álvaro Cartea
Time-series data in real-world medical settings typically exhibit long-range
dependencies and are observed at non-uniform intervals. In such contexts,
traditional sequence-based recurrent models struggle. To overcome this,
researchers replace recurrent architectures with Neural ODE-based models to
model irregularly sampled data and use Transformer-based architectures to
account for long-range dependencies. Despite the success of these two
approaches, both incur very high computational costs for input sequences of
moderate lengths and greater. To mitigate this, we introduce the Rough
Transformer, a variation of the Transformer model which operates on
continuous-time representations of input sequences and incurs significantly
reduced computational costs, critical for addressing long-range dependencies
common in medical contexts. In particular, we propose multi-view signature
attention, which uses path signatures to augment vanilla attention and to
capture both local and global dependencies in input data, while remaining
robust to changes in the sequence length and sampling frequency. We find that
Rough Transformers consistently outperform their vanilla attention counterparts
while obtaining the benefits of Neural ODE-based models using a fraction of the
computational time and memory resources on synthetic and real-world time-series
tasks.