Language Models Still Struggle to Zero-shot Reason about Time Series
Mike A. Merrill Mingtian Tan Vinayak Gupta Tom Hartvigsen Tim Althoff
Time series are critical for decision-making in fields like finance and
healthcare. Their importance has driven a recent influx of works passing time
series into language models, leading to non-trivial forecasting on some
datasets. But it remains unknown whether non-trivial forecasting implies that
language models can reason about time series. To address this gap, we generate
a first-of-its-kind evaluation framework for time series reasoning, including
formal tasks and a corresponding dataset of multi-scale time series paired with
text captions across ten domains. Using these data, we probe whether language
models achieve three forms of reasoning: (1) Etiological Reasoning - given an
input time series, can the language model identify the scenario that most
likely created it? (2) Question Answering - can a language model answer factual
questions about time series? (3) Context-Aided Forecasting - does highly
relevant textual context improve a language model's time series forecasts?
  We find that otherwise highly-capable language models demonstrate
surprisingly limited time series reasoning: they score marginally above random
on etiological and question answering tasks (up to 30 percentage points worse
than humans) and show modest success in using context to improve forecasting.
These weakness showcase that time series reasoning is an impactful, yet deeply
underdeveloped direction for language model research. We also make our datasets
and code public at to support further research in this direction at
https://github.com/behavioral-data/TSandLanguage