Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment
This work addresses the critical issue of durable and verifiable safety engineering in AI alignment, which is important for researchers and practitioners in AI safety, though it appears incremental by building on existing alignment methods.
The paper tackles the problem of AI alignment by addressing the structural flaw of entangling safety objectives with agent policies, which leads to opaque and single-use alignment artifacts. The result is a framework that decouples alignment artifact learning from policy optimization, producing an inspectable, editable, and model-agnostic reward model, and introduces a human-in-the-loop lifecycle to iteratively harden the reward model through automated audits and refinement.
AI alignment is growing in importance, yet current approaches suffer from a critical structural flaw that entangles the safety objectives with the agent's policy. Methods such as Reinforcement Learning from Human Feedback and Direct Preference Optimization create opaque, single-use alignment artifacts, which we term Alignment Waste. We propose Interactionless Inverse Reinforcement Learning to decouple alignment artifact learning from policy optimization, producing an inspectable, editable, and model-agnostic reward model. Additionally, we introduce the Alignment Flywheel, a human-in-the-loop lifecycle that iteratively hardens the reward model through automated audits and refinement. This architecture transforms safety from a disposable expense into a durable, verifiable engineering asset.