Spatiotemporal Representation Learning for Short and Long Medical Image
  Time Series
Chengzhi Shen Martin J. Menten Hrvoje Bogunović Ursula Schmidt-Erfurth Hendrik Scholl Sobha Sivaprasad Andrew Lotery Daniel Rueckert Paul Hager Robbie Holland
Analyzing temporal developments is crucial for the accurate prognosis of many
medical conditions. Temporal changes that occur over short time scales are key
to assessing the health of physiological functions, such as the cardiac cycle.
Moreover, tracking longer term developments that occur over months or years in
evolving processes, such as age-related macular degeneration (AMD), is
essential for accurate prognosis. Despite the importance of both short and long
term analysis to clinical decision making, they remain understudied in medical
deep learning. State of the art methods for spatiotemporal representation
learning, developed for short natural videos, prioritize the detection of
temporal constants rather than temporal developments. Moreover, they do not
account for varying time intervals between acquisitions, which are essential
for contextualizing observed changes. To address these issues, we propose two
approaches. First, we combine clip-level contrastive learning with a novel
temporal embedding to adapt to irregular time series. Second, we propose
masking and predicting latent frame representations of the temporal sequence.
Our two approaches outperform all prior methods on temporally-dependent tasks
including cardiac output estimation and three prognostic AMD tasks. Overall,
this enables the automated analysis of temporal patterns which are typically
overlooked in applications of deep learning to medicine.