Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective
This addresses the challenge of performance tuning for software engineers by providing a novel perspective, though it is incremental as it builds on existing fitness landscape concepts.
The paper tackled the problem of analyzing performance in configurable software systems by modeling the configuration space as a fitness landscape, using a graph data mining framework to analyze 86 million configurations from real-world systems, revealing 6 key findings about landscape topography.
Modern software systems are often highly configurable to tailor varied requirements from diverse stakeholders. Understanding the mapping between configurations and the desired performance attributes plays a fundamental role in advancing the controllability and tuning of the underlying system, yet has long been a dark hole of knowledge due to its black-box nature. While there have been previous efforts in performance analysis for these systems, they analyze the configurations as isolated data points without considering their inherent spatial relationships. This renders them incapable of interrogating many important aspects of the configuration space like local optima. In this work, we advocate a novel perspective to rethink performance analysis -- modeling the configuration space as a structured ``landscape''. To support this proposition, we designed \our, an open-source, graph data mining empowered fitness landscape analysis (FLA) framework. By applying this framework to $86$M benchmarked configurations from $32$ running workloads of $3$ real-world systems, we arrived at $6$ main findings, which together constitute a holistic picture of the landscape topography, with thorough discussions about their implications on both configuration tuning and performance modeling.