Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL
This work addresses interpretability and safety challenges in LLM alignment for researchers and practitioners, offering an incremental improvement over standard IRL methods.
The paper tackles the problem of understanding latent reward signals in RLHF-aligned LLMs by introducing a failure-aware IRL algorithm that focuses on misclassified or difficult examples to recover these rewards. It demonstrates that this method outperforms existing IRL baselines in LLM detoxification tasks, enabling more effective re-RLHF training without external supervision.
Reinforcement Learning from Human Feedback (RLHF) aligns Large Language Models (LLMs) with human preferences, yet the underlying reward signals they internalize remain hidden, posing a critical challenge for interpretability and safety. Existing approaches attempt to extract these latent incentives using Inverse Reinforcement Learning (IRL), but treat all preference pairs equally, often overlooking the most informative signals: those examples the extracted reward model misclassifies or assigns nearly equal scores, which we term \emph{failures}. We introduce a novel \emph{failure-aware} IRL algorithm that focuses on misclassified or difficult examples to recover the latent rewards defining model behaviors. By learning from these failures, our failure-aware IRL extracts reward functions that better reflect the true objectives behind RLHF. We demonstrate that failure-aware IRL outperforms existing IRL baselines across multiple metrics when applied to LLM detoxification, without requiring external classifiers or supervision. Crucially, failure-aware IRL yields rewards that better capture the true incentives learned during RLHF, enabling more effective re-RLHF training than standard IRL. This establishes failure-aware IRL as a robust, scalable method for auditing model alignment and reducing ambiguity in the IRL process.