Towards Comparability in Non-Intrusive Load Monitoring: On Data and Performance Evaluation
This addresses reproducibility and comparability challenges for researchers in energy disaggregation, but it is incremental as it focuses on evaluation rather than new methods.
The paper tackles the problem of non-comparability in Non-Intrusive Load Monitoring (NILM) research due to lack of standardization in datasets and evaluation procedures, and it highlights urgent issues such as data pre-processing and unified performance reporting without providing specific numerical results.
Non-Intrusive Load Monitoring (NILM) comprises of a set of techniques that provide insights into the energy consumption of households and industrial facilities. Latest contributions show significant improvements in terms of accuracy and generalisation abilities. Despite all progress made concerning disaggregation techniques, performance evaluation and comparability remains an open research question. The lack of standardisation and consensus on evaluation procedures makes reproducibility and comparability extremely difficult. In this paper, we draw attention to comparability in NILM with a focus on highlighting the considerable differences amongst common energy datasets used to test the performance of algorithms. We divide discussion on comparability into data aspects, performance metrics, and give a close view on evaluation processes. Detailed information on pre-processing as well as data cleaning methods, the importance of unified performance reporting, and the need for complexity measures in load disaggregation are found to be the most urgent issues in NILM-related research. In addition, our evaluation suggests that datasets should be chosen carefully. We conclude by formulating suggestions for future work to enhance comparability.