Evaluating the Safety and Skill Reasoning of Large Reasoning Models Under Compute Constraints
This work addresses computational efficiency and safety trade-offs for users of large reasoning models, but it is incremental as it builds on existing test-time compute scaling methods.
The paper tackles the problem of high computational costs in reasoning language models by investigating compute constraint strategies like reasoning length constraints and quantization, and finds that these methods can reduce compute demand while studying their impact on safety performance.
Test-time compute scaling has demonstrated the ability to improve the performance of reasoning language models by generating longer chain-of-thought (CoT) sequences. However, this increase in performance comes with a significant increase in computational cost. In this work, we investigate two compute constraint strategies: (1) reasoning length constraint and (2) model quantization, as methods to reduce the compute demand of reasoning models and study their impact on their safety performance. Specifically, we explore two approaches to apply compute constraints to reasoning models: (1) fine-tuning reasoning models using a length controlled policy optimization (LCPO) based reinforcement learning method to satisfy a user-defined CoT reasoning length, and (2) applying quantization to maximize the generation of CoT sequences within a user-defined compute constraint. Furthermore, we study the trade-off between the computational efficiency and the safety of the model.