AURA: Affordance-Understanding and Risk-aware Alignment Technique for Large Language Models
This addresses safety risks for users of LLMs by providing a more granular and proactive alignment technique, though it appears incremental as it builds on existing reward modeling approaches.
The paper tackles the problem of affordance-based safety risks in large language models, where outputs inadvertently facilitate harmful actions, by introducing AURA, a multi-layered framework using Process Reward Models that significantly improves logical integrity and safety in model outputs.
Present day LLMs face the challenge of managing affordance-based safety risks-situations where outputs inadvertently facilitate harmful actions due to overlooked logical implications. Traditional safety solutions, such as scalar outcome-based reward models, parameter tuning, or heuristic decoding strategies, lack the granularity and proactive nature needed to reliably detect and intervene during subtle yet crucial reasoning steps. Addressing this fundamental gap, we introduce AURA, an innovative, multi-layered framework centered around Process Reward Models (PRMs), providing comprehensive, step level evaluations across logical coherence and safety-awareness. Our framework seamlessly combines introspective self-critique, fine-grained PRM assessments, and adaptive safety-aware decoding to dynamically and proactively guide models toward safer reasoning trajectories. Empirical evidence clearly demonstrates that this approach significantly surpasses existing methods, significantly improving the logical integrity and affordance-sensitive safety of model outputs. This research represents a pivotal step toward safer, more responsible, and contextually aware AI, setting a new benchmark for alignment-sensitive applications.