RAS: Measuring LLM Safety Through Refusal Alignment
For LLM safety evaluators, this provides a faster, cheaper alternative to output-level safety testing, though it is an incremental improvement over existing representation-based methods.
The paper proposes SafeVec, a white-box evaluation method that measures LLM safety by analyzing internal representations rather than generated outputs. RAS scores reliably separate aligned from unsafe models across Llama, Gemma, and Qwen families, track attack success rates, and are faster than judge-based evaluation.
Safety evaluation of large language models (LLMs) is commonly performed by querying models with unsafe or jailbreak prompts and judging whether their outputs violate a safety policy. Although useful, output-level evaluation is expensive, sensitive to judge choice, and easily tied to fixed question banks. We propose **SafeVec**, a white-box evaluation procedure that measures safety from internal representations rather than generated answers. **SafeVec** first extracts layer-wise refusal directions from a safety-aligned reference model, then selects stable layer windows where safe and unsafe behaviors are separable, and finally scores a target model by measuring whether its hidden states align with these refusal directions under unsafe and jailbreak prompts. The resulting metric, **RAS** (**R**efusal **A**lignment **S**core), maps representation-level refusal alignment to a calibrated 0-100 safety score. Across `Llama`, `Gemma`, and `Qwen` model families, RAS separates aligned models from uncensored and abliterated variants, tracks output-level attack success rate, and is substantially faster than judge-based evaluation. These results suggest that refusal alignment provides a compact and efficient signal for white-box LLM safety evaluation.