A Blind Spot in Alignment: Quantifying Biosecurity Risks in Large Language Models
This work addresses a critical blind spot in LLM safety evaluation by providing a domain-specific benchmark and metric for biosecurity, which is important for AI safety researchers and policymakers, though it is domain-specific and incremental in methodology.
The paper introduces SPIKE-Bench, a benchmark with 631 toxin-design prompts and a three-stage filtering protocol, to evaluate biosecurity risks in LLMs. Auditing 32 LLMs, they find most comply with toxin-design requests, with a Functional Harmfulness Rate reaching 50.7%, and that refusal rate does not predict functional risk. They also provide BioSafe-Guard, a classifier that reduces predicted functional risk while preserving benign utility.
Large Language Models (LLMs) are accelerating biological research, yet this same capability poses a critical biosecurity threat: models that assist in protein engineering can equally be prompted to generate predicted toxin-like sequences, potentially lowering the barrier to biological misuse. Current safety evaluations, however, operate in natural language and cannot determine whether a model-generated amino acid sequence is biological gibberish or a computational risk signal. To address this evaluation blind spot, we introduce SPIKE-Bench, coupling 631 curated toxin-design prompts across seven functional categories with the SPIKE funnel, a three-stage protocol that filters output through compliance, biological plausibility, and predicted toxicity, producing stage-level diagnostics and an aggregate function-aware metric: the Functional Harmfulness Rate (FHR). An audit of 32 LLMs reveals that most models freely comply with toxin-design requests; FHR is driven primarily by biological generation capability rather than safety alignment, reaching 50.7%; and Refusal Rate fails to predict functional risk. As a first step toward mitigation, we provide BioSafe-Guard, a domain-specialized classifier that substantially reduces predicted functional risk while preserving benign utility. We release SPIKE-Bench and BioSafe-Guard at https://github.com/PKU-Alignment/SPIKE-Bench to support more rigorous biosecurity evaluation of LLMs.