Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimes
For developers of LLM safety classifiers, this work provides a compact supervision signal that improves robustness across training regimes, though the dataset is small and results may be incremental.
The authors propose intent-aware training for safety classifiers, introducing the AIMS dataset of 1,724 prompts with intent annotations. They show that intent-aware methods, especially GRPO rewarding intent faithfulness, achieve strong performance across five safety benchmarks, forming the latency-F1 Pareto frontier.
We argue that safety classifiers should model user intent as an explicit signal between the prompt and the final label. To study this, we introduce AIMS, a human-annotated dataset of 1,724 difficult safety prompts, each paired with an intent description and harm label. We use AIMS to evaluate intent-aware training across supervised fine-tuning, preference learning, reasoning distillation, and reinforcement learning. Despite its size, AIMS enables competitive safety classifiers across training regimes: DPO from model-generated intent errors improves over SFT, and intent-conditioned distillation outperforms reasoning-only distillation in most teacher-student pairs. Most notably, directly rewarding intent faithfulness with GRPO yields the strongest average performance across five external safety benchmarks, while our intent-aware models form the inference latency-F1 Pareto frontier. These results show that faithful intent modeling is a compact, high-quality supervision signal for more robust safety classifiers.