STEB: In Search of the Best Evaluation Approach for Synthetic Time Series
This addresses the need for standardized evaluation in synthetic time series generation, which is crucial for data augmentation and privacy applications, but it is incremental as it builds on existing measures.
The authors tackled the challenge of objectively comparing evaluation measures for synthetic time series by proposing STEB, the first benchmark framework for automated comparisons, and used it to rank 41 measures, finding that the choice of time series embedding significantly affects scores.
The growing need for synthetic time series, due to data augmentation or privacy regulations, has led to numerous generative models, frameworks, and evaluation measures alike. Objectively comparing these measures on a large scale remains an open challenge. We propose the Synthetic Time series Evaluation Benchmark (STEB) -- the first benchmark framework that enables comprehensive and interpretable automated comparisons of synthetic time series evaluation measures. Using 10 diverse datasets, randomness injection, and 13 configurable data transformations, STEB computes indicators for measure reliability and score consistency. It tracks running time, test errors, and features sequential and parallel modes of operation. In our experiments, we determine a ranking of 41 measures from literature and confirm that the choice of upstream time series embedding heavily impacts the final score.