Bidirectional Generative Pre-training for Improving Healthcare Time-series Representation Learning
This work addresses the problem of limited pre-training methods for healthcare time-series data, offering a domain-specific improvement for medical applications.
The paper tackles the challenge of learning time-series representations for healthcare tasks by proposing BiTimelyGPT, a novel architecture that pre-trains on biosignals and clinical records using bidirectional token prediction, resulting in superior performance in predicting neurological functionality, disease diagnosis, and physiological signs.
Learning time-series representations for discriminative tasks, such as classification and regression, has been a long-standing challenge in the healthcare domain. Current pre-training methods are limited in either unidirectional next-token prediction or randomly masked token prediction. We propose a novel architecture called Bidirectional Timely Generative Pre-trained Transformer (BiTimelyGPT), which pre-trains on biosignals and longitudinal clinical records by both next-token and previous-token prediction in alternating transformer layers. This pre-training task preserves original distribution and data shapes of the time-series. Additionally, the full-rank forward and backward attention matrices exhibit more expressive representation capabilities. Using biosignals and longitudinal clinical records, BiTimelyGPT demonstrates superior performance in predicting neurological functionality, disease diagnosis, and physiological signs. By visualizing the attention heatmap, we observe that the pre-trained BiTimelyGPT can identify discriminative segments from biosignal time-series sequences, even more so after fine-tuning on the task.