AttackonCTF: Defending Hardware Security Competition Benchmarks in the Age of LLMs
For organizers of hardware security competitions, this work addresses a critical validity threat from LLMs by providing a practical obfuscation method.
LLMs exploit syntactic diff patterns in hardware security competition benchmarks, achieving 83% detection without genuine reasoning. The authors propose a semantics-preserving obfuscation framework that reduces LLM detection accuracy by 50% with 10% obfuscation and by 78.6% under full obfuscation, restoring benchmark validity.
Hardware security competitions such as HackTheSilicon serve as benchmarking platforms for evaluating vulnerability detection methods and for training humans and AI. However, our study reveals that LLMs threaten their validity. Instead of genuine security reasoning, detectors exploit a diff-style syntactic comparison, achieving an 83% detection rate, undermining fair evaluation. To mitigate this, we propose the first LLM-oriented, semantics-preserving obfuscation framework for these benchmarks. Unlike IP-protection approaches, it applies human-readable transformations and controlled diff-noise while preserving functionality. On HackTheSilicon, the framework reduces LLM-based detection accuracy by 50% with only 10% obfuscation and by 78.6% under complete obfuscation, restoring benchmark reliability.