fev-bench: A Realistic Benchmark for Time Series Forecasting
This addresses the need for realistic and statistically rigorous benchmarks in time series forecasting, particularly for researchers and practitioners evaluating pretrained models, though it is incremental as it builds on existing benchmarking concepts.
The authors tackled the problem of inadequate benchmarks for time series forecasting by introducing fev-bench, a comprehensive benchmark with 100 tasks across seven domains including 46 with covariates, and fev, a Python library for reproducible evaluation. They reported results using principled aggregation methods with bootstrapped confidence intervals for various models, identifying future research directions.
Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly given the recent rise of pretrained models. Existing benchmarks often have narrow domain coverage or overlook important real-world settings, such as tasks with covariates. Additionally, their aggregation procedures often lack statistical rigor, making it unclear whether observed performance differences reflect true improvements or random variation. Many benchmarks also fail to provide infrastructure for consistent evaluation or are too rigid to integrate into existing pipelines. To address these gaps, we propose fev-bench, a benchmark comprising 100 forecasting tasks across seven domains, including 46 tasks with covariates. Supporting the benchmark, we introduce fev, a lightweight Python library for benchmarking forecasting models that emphasizes reproducibility and seamless integration with existing workflows. Usingfev, fev-bench employs principled aggregation methods with bootstrapped confidence intervals to report model performance along two complementary dimensions: win rates and skill scores. We report results on fev-bench for various pretrained, statistical and baseline models, and identify promising directions for future research.