Unicorn: Reasoning about Configurable System Performance through the lens of Causality
This addresses performance debugging and optimization for configurable systems like ML systems and databases, offering a novel approach to improve reliability in unseen environments.
The paper tackled the challenge of understanding performance in highly configurable computer systems by proposing Unicorn, a method that uses causal inference to model interactions and predict performance, resulting in outperforming state-of-the-art methods in finding effective repairs and near-optimal configurations across six systems.
Modern computer systems are highly configurable, with the total variability space sometimes larger than the number of atoms in the universe. Understanding and reasoning about the performance behavior of highly configurable systems, over a vast and variable space, is challenging. State-of-the-art methods for performance modeling and analyses rely on predictive machine learning models, therefore, they become (i) unreliable in unseen environments (e.g., different hardware, workloads), and (ii) may produce incorrect explanations. To tackle this, we propose a new method, called Unicorn, which (i) captures intricate interactions between configuration options across the software-hardware stack and (ii) describes how such interactions can impact performance variations via causal inference. We evaluated Unicorn on six highly configurable systems, including three on-device machine learning systems, a video encoder, a database management system, and a data analytics pipeline. The experimental results indicate that Unicorn outperforms state-of-the-art performance debugging and optimization methods in finding effective repairs for performance faults and finding configurations with near-optimal performance. Further, unlike the existing methods, the learned causal performance models reliably predict performance for new environments.