Spatiotemporal Representation Learning for Short and Long Medical Image Time Series
This work addresses the challenge of automated temporal analysis in medical imaging for improved clinical decision-making, representing an incremental advance in adapting video-based methods to irregular medical time series.
The paper tackled the problem of analyzing both short and long-term temporal developments in medical images, which are crucial for prognosis but understudied in deep learning, by proposing two spatiotemporal representation learning approaches that outperform prior methods on tasks like cardiac output estimation and AMD prognosis.
Analyzing temporal developments is crucial for the accurate prognosis of many medical conditions. Temporal changes that occur over short time scales are key to assessing the health of physiological functions, such as the cardiac cycle. Moreover, tracking longer term developments that occur over months or years in evolving processes, such as age-related macular degeneration (AMD), is essential for accurate prognosis. Despite the importance of both short and long term analysis to clinical decision making, they remain understudied in medical deep learning. State of the art methods for spatiotemporal representation learning, developed for short natural videos, prioritize the detection of temporal constants rather than temporal developments. Moreover, they do not account for varying time intervals between acquisitions, which are essential for contextualizing observed changes. To address these issues, we propose two approaches. First, we combine clip-level contrastive learning with a novel temporal embedding to adapt to irregular time series. Second, we propose masking and predicting latent frame representations of the temporal sequence. Our two approaches outperform all prior methods on temporally-dependent tasks including cardiac output estimation and three prognostic AMD tasks. Overall, this enables the automated analysis of temporal patterns which are typically overlooked in applications of deep learning to medicine.