Semantics-Preserving Evasion of LLM Vulnerability Detectors
This reveals a critical security flaw in LLM-based code review systems, posing risks for deployment in security-critical applications.
The study found that state-of-the-art LLM-based vulnerability detectors for C/C++ code are highly susceptible to evasion through semantics-preserving edits, with predictions flipping under behavior-equivalent transformations, and universal adversarial strings effectively transferring to black-box APIs.
LLM-based vulnerability detectors are increasingly deployed in security-critical code review, yet their resilience to evasion under behavior-preserving edits remains poorly understood. We evaluate detection-time integrity under a semantics-preserving threat model by instantiating diverse behavior-preserving code transformations on a unified C/C++ benchmark (N=5000), and introduce a metric of joint robustness across different attack methods/carriers. Across models, we observe a systemic failure of semantic invariant adversarial transformations: even state-of-the-art vulnerability detectors perform well on clean inputs while predictions flip under behavior-equivalent edits. Universal adversarial strings optimized on a single surrogate model remain effective when transferred to black-box APIs, and gradient access can further amplify evasion success. These results show that even high-performing detectors are vulnerable to low-cost, semantics-preserving evasion. Our carrier-based metrics provide practical diagnostics for evaluating LLM-based code detectors.