Xiaolei Ren

Xiaolei Ren

Prior work has demonstrated that functionally correct yet vulnerable outputs arise systematically in threat-oriented settings, where adversarial or implicit channels are used to induce security failures in code agents and automated patching workflows. This note introduces a complementary but distinct framing: False Security Confidence (FSC), which studies the same surface phenomenon from a measurement-first perspective in ordinary, non-attack-framed generation tasks. Our interest is not in whether attacks can produce such outputs, but in how frequently and in what forms they appear absent explicit attack pressure, and whether conventional functional evaluation reliably detects them. We formalize FSC rate as the prevalence of security failure within the set of functionally correct outputs, distinguish it from prior joint functional-security metrics such as SAFE and outcome-driven evaluation frameworks such as CWEval, define a three-ecosystem task view for studying how FSC manifests across general-purpose programming, deployment-context tasks, and security-explicit programming, and identify FSC-hard as a practically important refinement layer in which static analyzers miss vulnerabilities that remain dynamically triggerable. This technical report is intentionally scoped as a framework statement rather than a full empirical paper: its purpose is to establish terminology, measurement boundaries, and study design commitments for subsequent large-scale evaluation.

Xiaolei Ren, Michael Ho, Jiang Ming et al.

Since compiler optimization is the most common source contributing to binary code differences in syntax, testing the resilience against the changes caused by different compiler optimization settings has become a standard evaluation step for most binary diffing approaches. For example, 47 top-venue papers in the last 12 years compared different program versions compiled by default optimization levels (e.g., -Ox in GCC and LLVM). Although many of them claim they are immune to compiler transformations, it is yet unclear about their resistance to non-default optimization settings. Especially, we have observed that adversaries explored non-default compiler settings to amplify malware differences. This paper takes the first step to systematically studying the effectiveness of compiler optimization on binary code differences. We tailor search-based iterative compilation for the auto-tuning of binary code differences. We develop BinTuner to search near-optimal optimization sequences that can maximize the amount of binary code differences. We run BinTuner with GCC 10.2 and LLVM 11.0 on SPEC benchmarks (CPU2006 & CPU2017), Coreutils, and OpenSSL. Our experiments show that at the cost of 279 to 1,881 compilation iterations, BinTuner can find custom optimization sequences that are substantially better than the general -Ox settings. BinTuner's outputs seriously undermine prominent binary diffing tools' comparisons. In addition, the detection rate of the IoT malware variants tuned by BinTuner falls by more than 50%. Our findings paint a cautionary tale for security analysts that attackers have a new way to mutate malware code cost-effectively, and the research community needs to step back to reassess optimization-resistance evaluations.