Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data
This work addresses the lack of multimodal datasets for forecasting in domains like climate science and healthcare, though it is incremental as the model did not achieve improvements.
The authors tackled the problem of multimodal forecasting by creating the TimeText Corpus (TTC), a curated dataset with aligned time series and textual data from climate science and healthcare, but their proposed Hybrid Multi-Modal Forecaster (Hybrid-MMF) model did not outperform existing baselines in experiments.
Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.