WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series
This addresses the problem of evaluating OOD generalization for time series tasks, which is underexplored compared to static computer vision, by providing new benchmarks for researchers.
The authors tackled the lack of benchmarks for out-of-distribution generalization in time series by introducing WOODS, a set of eight open-source benchmarks covering diverse data modalities like videos and sensor signals, and found that existing algorithms show large room for improvement on these datasets.
Machine learning models often fail to generalize well under distributional shifts. Understanding and overcoming these failures have led to a research field of Out-of-Distribution (OOD) generalization. Despite being extensively studied for static computer vision tasks, OOD generalization has been underexplored for time series tasks. To shine light on this gap, we present WOODS: eight challenging open-source time series benchmarks covering a diverse range of data modalities, such as videos, brain recordings, and sensor signals. We revise the existing OOD generalization algorithms for time series tasks and evaluate them using our systematic framework. Our experiments show a large room for improvement for empirical risk minimization and OOD generalization algorithms on our datasets, thus underscoring the new challenges posed by time series tasks. Code and documentation are available at https://woods-benchmarks.github.io .