Annotating the Chain-of-Thought: A Behavior-Labeled Dataset for AI Safety
This addresses AI safety for researchers and practitioners by providing a dataset to enhance monitoring techniques, though it is incremental as it builds on existing activation-based methods.
The paper tackles the problem of monitoring chain-of-thought reasoning for AI safety by presenting a sentence-level labeled dataset that enables activation-based detection and steering of safety behaviors, demonstrating its utility in improving safety oversight.
Recent work has highlighted the importance of monitoring chain-of-thought reasoning for AI safety; however, current approaches that analyze textual reasoning steps can miss subtle harmful patterns and may be circumvented by models that hide unsafe reasoning. We present a sentence-level labeled dataset that enables activation-based monitoring of safety behaviors during LLM reasoning. Our dataset contains reasoning sequences with sentence-level annotations of safety behaviors such as expression of safety concerns or speculation on user intent, which we use to extract steering vectors for detecting and influencing these behaviors within model activations. The dataset fills a key gap in safety research: while existing datasets label reasoning holistically, effective application of steering vectors for safety monitoring could be improved by identifying precisely when specific behaviors occur within reasoning chains. We demonstrate the dataset's utility by extracting representations that both detect and steer safety behaviors in model activations, showcasing the potential of activation-level techniques for improving safety oversight on reasoning. Content Warning: This paper discusses AI safety in the context of harmful prompts and may contain references to potentially harmful content.