Towards Rigorous Validation of Energy Optimisation Experiments
This addresses the need for rigorous validation in energy optimization for practitioners in Search-Based Software Engineering and Genetic Improvement, though it appears incremental as it builds on existing validation methods.
The paper tackled the problem of noisy and expensive fitness evaluations in software energy optimization by proposing R3-validation, a rotated-round-robin approach, which was shown to better align with actual platform behavior across multiple devices and operating systems.
The optimisation of software energy consumption is of growing importance across all scales of modern computing, i.e., from embedded systems to data-centres. Practitioners in the field of Search-Based Software Engineering and Genetic Improvement of Software acknowledge that optimising software energy consumption is difficult due to noisy and expensive fitness evaluations. However, it is apparent from results to date that more progress needs to be made in rigorously validating optimisation results. This problem is pressing because modern computing platforms have highly complex and variable behaviour with respect to energy consumption. To compare solutions fairly we propose in this paper a new validation approach called R3-validation which exercises software variants in a rotated-round-robin order. Using a case study, we present an in-depth analysis of the impacts of changing system states on software energy usage, and we show how R3-validation mitigates these. We compare it with current validation approaches across multiple devices and operating systems, and we show that it aligns better with actual platform behaviour.