Holistic Measures for Evaluating Prediction Models in Smart Grids
This work addresses the need for application-sensitive evaluation metrics for prediction models in smart grids, benefiting data mining researchers and practitioners, but it is incremental as it builds on existing evaluation concepts.
The paper tackles the problem of evaluating prediction models in smart grids by proposing holistic performance measures that go beyond traditional error metrics, and demonstrates their value through empirical analysis on real energy data for applications like planning and demand response.
The performance of prediction models is often based on "abstract metrics" that estimate the model's ability to limit residual errors between the observed and predicted values. However, meaningful evaluation and selection of prediction models for end-user domains requires holistic and application-sensitive performance measures. Inspired by energy consumption prediction models used in the emerging "big data" domain of Smart Power Grids, we propose a suite of performance measures to rationally compare models along the dimensions of scale independence, reliability, volatility and cost. We include both application independent and dependent measures, the latter parameterized to allow customization by domain experts to fit their scenario. While our measures are generalizable to other domains, we offer an empirical analysis using real energy use data for three Smart Grid applications: planning, customer education and demand response, which are relevant for energy sustainability. Our results underscore the value of the proposed measures to offer a deeper insight into models' behavior and their impact on real applications, which benefit both data mining researchers and practitioners.