Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment
This work addresses the problem of style-conditioned policy learning for researchers in offline RL, though it appears incremental as it builds on existing techniques.
The paper tackles the challenge of aligning style with high task performance in offline reinforcement learning by proposing a unified definition of behavior style and introducing SCIQL, a framework that achieves superior performance on both objectives compared to prior methods.
We study offline reinforcement learning of style-conditioned policies using explicit style supervision via subtrajectory labeling functions. In this setting, aligning style with high task performance is particularly challenging due to distribution shift and inherent conflicts between style and reward. Existing methods, despite introducing numerous definitions of style, often fail to reconcile these objectives effectively. To address these challenges, we propose a unified definition of behavior style and instantiate it into a practical framework. Building on this, we introduce Style-Conditioned Implicit Q-Learning (SCIQL), which leverages offline goal-conditioned RL techniques, such as hindsight relabeling and value learning, and combine it with a new Gated Advantage Weighted Regression mechanism to efficiently optimize task performance while preserving style alignment. Experiments demonstrate that SCIQL achieves superior performance on both objectives compared to prior offline methods. Code, datasets and visuals are available in: https://sciql-iclr-2026.github.io/.