CLLMate: A Multimodal Benchmark for Weather and Climate Events Forecasting
This work addresses the gap in environmental forecasting by enabling actionable event predictions, which is crucial for mitigating hazards, though it is incremental as it builds on existing multimodal methods.
The authors tackled the problem of forecasting weather and climate events by proposing a new task that uses numerical and textual data to predict events, and they introduced CLLMate, a multimodal dataset with 26,156 articles aligned with ERA5 data, benchmarking 23 models to reveal their advantages and limitations.
Forecasting weather and climate events is crucial for making appropriate measures to mitigate environmental hazards and minimize losses. However, existing environmental forecasting research focuses narrowly on predicting numerical meteorological variables (e.g., temperature), neglecting the translation of these variables into actionable textual narratives of events and their consequences. To bridge this gap, we proposed Weather and Climate Event Forecasting (WCEF), a new task that leverages numerical meteorological raster data and textual event data to predict weather and climate events. This task is challenging to accomplish due to difficulties in aligning multimodal data and the lack of supervised datasets. To address these challenges, we present CLLMate, the first multimodal dataset for WCEF, using 26,156 environmental news articles aligned with ERA5 reanalysis data. We systematically benchmark 23 existing MLLMs on CLLMate, including closed-source, open-source, and our fine-tuned models. Our experiments reveal the advantages and limitations of existing MLLMs and the value of CLLMate for the training and benchmarking of the WCEF task.