Unlocking the Secrets of Software Configuration Landscapes-Ruggedness, Accessibility, Escapability, and Transferability
This work addresses the challenge of tuning configurable software systems for diverse stakeholders, providing foundational insights for designing tailored meta-heuristics, though it is incremental in applying existing fitness landscape analysis methods to a new domain.
The paper tackled the problem of understanding the complex mapping between configurations and performance in configurable software systems by conducting a fitness landscape analysis on 86 million evaluated configurations across three real-world systems and 32 workloads, revealing that software configuration landscapes are rugged with many local optima but top optima are highly accessible and inferior ones are escapable, with structural similarities across workloads that can expedite search.
Modern software systems are often highly configurable to tailor varied requirements from diverse stakeholders. Understanding the mapping between configurations and the desired performance attributes plays a fundamental role in advancing the controllability and tuning of the underlying system, yet has long been a dark hole of knowledge due to their black-box nature and the enormous combinatorial configuration space. In this paper, using $86$M evaluated configurations from three real-world systems on $32$ running workloads, we conducted one of its kind fitness landscape analysis (FLA) for configurable software systems. With comprehensive FLA methods, we for the first time show that: $i)$ the software configuration landscapes are fairly rugged, with numerous scattered local optima; $ii)$ nevertheless, the top local optima are highly accessible, featuring significantly larger basins of attraction; $iii)$ most inferior local optima are escapable with simple perturbations; $iv)$ landscapes of the same system with different workloads share structural similarities, which can be exploited to expedite heuristic search. Our results also provide valuable insights on the design of tailored meta-heuristics for configuration tuning; our FLA framework along with the collected data, build solid foundation for future research in this direction.