Evaluating Time Series Models with Knowledge Discovery
This is an incremental blue-sky idea for time series data mining in domains like healthcare and energy, addressing evaluation challenges but without implementation or results.
The paper tackles the problem that standard metric-based evaluation (e.g., RMSE, Accuracy) for time series deep learning models may not reflect real-world performance due to data complexities like unknown environmental factors and non-stationary behavior, and proposes a knowledge-discovery-based evaluation framework using domain expertise to improve generalization and persuasive power.
Time series data is one of the most ubiquitous data modalities existing in a diverse critical domains such as healthcare, seismology, manufacturing and energy. Recent years, there are increasing interest of the data mining community to develop time series deep learning models to pursue better performance. The models performance often evaluate by certain evaluation metrics such as RMSE, Accuracy, and F1-score. Yet time series data are often hard to interpret and are collected with unknown environmental factors, sensor configuration, latent physic mechanisms, and non-stationary evolving behavior. As a result, a model that is better on standard metric-based evaluation may not always perform better in real-world tasks. In this blue sky paper, we aim to explore the challenge that exists in the metric-based evaluation framework for time series data mining and propose a potential blue-sky idea -- developing a knowledge-discovery-based evaluation framework, which aims to effectively utilize domain-expertise knowledge to evaluate a model. We demonstrate that an evidence-seeking explanation can potentially have stronger persuasive power than metric-based evaluation and obtain better generalization ability for time series data mining tasks.