Symetra: Visual Analytics for the Parameter Tuning Process of Symbolic Execution Engines
This addresses the difficulty for software testing experts in optimizing symbolic execution engines, offering a human-in-the-loop solution that is incremental over existing automated tuners.
The paper tackles the problem of tuning parameters in symbolic execution engines like KLEE, which is challenging due to many parameters and limited insights from automated methods, and presents Symetra, a visual analytics system that enables human-in-the-loop tuning, resulting in improved branch coverage and tuning efficiency over automated approaches.
Symbolic execution engines such as KLEE automatically generate test cases to maximize branch coverage, but their numerous parameters make it difficult to understand the parameters' impact, leading the user to rely on suboptimal default configurations. While automated tuners have shown promising results, they provide limited insights into why certain configurations work well, motivating the need for Human-in-the-Loop approaches. In this work, we present a visual analytics system, Symetra, designed to support Human-in-the-Loop parameter tuning of symbolic execution engines. To handle a large number of parameters and their configurations, we provide two complementary overviews of their impact on branch coverage values and patterns. Building on these overviews, our system enables collective analysis, allowing the user to contrast groups of configurations and identify differences that may affect branch coverage. We also report on case studies and a Human-in-the-Loop tuning process, demonstrating that experts not only interpreted parameter impacts and identified complementary configurations, but also improved upon fully automated approaches in both branch coverage and tuning efficiency.