Not All Refusals Are Equal: How Safety Alignment Fails Cybersecurity at Scale
For cybersecurity practitioners and LLM developers, this work reveals a critical limitation of current safety alignment and provides a method to selectively remove refusal for legitimate cybersecurity operations.
The authors show that safety alignment in LLMs fails for cybersecurity tasks because it does not distinguish domains, and they achieve domain-specific abliteration on 24 open-source LLMs including a 1T-parameter model, demonstrating that refusal subspaces are distributed across layers.
There is no doubt that safety alignment is an essential step in LLM training. However, conceptually it does not distinguish between various domains and the level of potential harm of a query, which creates significant complications in the fields like cyber security, where a model should not be constrained by its safety circuits to accomplish the goals of legitimate, authorized operations. In this work, we share our findings from a large scale abliteration experiment on 24 open-source LLMs and show that domain-specific abliteration is achievable with standard methodology on the example of a 1T-parameter Kimi K2. Building on recent work showing that refusal in LLMs occupies a multi-dimensional subspace within layers, we find that it is also distributed widely across layers, especially in trillion-parameter MoE architectures, and so we aim to capture the part of it that represents harmful concepts in the cybersecurity domain exclusively. We also investigate the correlation between models' features and the effect of domain-specific abliteration, identifying that the type of safety training and architecture are the most reliable predictors. Finally, we classify the models into 3 \emph{abliteration susceptibility} tiers and put forward a set of conjectures as to why a particular effect from this intervention might be observed in a given model.