Faster Discovery of Faster System Configurations with Spectral Learning
This work addresses the challenge for software engineers in optimizing system configurations with reduced sampling and improved stability, representing a strong specific gain in a domain-specific area.
The paper tackles the problem of inefficient performance prediction for configurable software systems by introducing the WHAT spectral learner, which uses spectral analysis for dimensionality reduction to reduce required samples and prediction variance, achieving less than 10% error with 2 to 10 times fewer samples than prior methods.
Despite the huge spread and economical importance of configurable software systems, there is unsatisfactory support in utilizing the full potential of these systems with respect to finding performance-optimal configurations. Prior work on predicting the performance of software configurations suffered from either (a) requiring far too many sample configurations or (b) large variances in their predictions. Both these problems can be avoided using the WHAT spectral learner. WHAT's innovation is the use of the spectrum (eigenvalues) of the distance matrix between the configurations of a configurable software system, to perform dimensionality reduction. Within that reduced configuration space, many closely associated configurations can be studied by executing only a few sample configurations. For the subject systems studied here, a few dozen samples yield accurate and stable predictors - less than 10% prediction error, with a standard deviation of less than 2%. When compared to the state of the art, WHAT (a) requires 2 to 10 times fewer samples to achieve similar prediction accuracies, and (b) its predictions are more stable (i.e., have lower standard deviation). Furthermore, we demonstrate that predictive models generated by WHAT can be used by optimizers to discover system configurations that closely approach the optimal performance.