Beyond Imitation: Recovering Dense Rewards from Demonstrations
This reframes SFT as a reward learning mechanism, offering new ways to leverage expert demonstrations for policy improvement in AI.
The paper establishes that supervised fine-tuning (SFT) is equivalent to inverse reinforcement learning, revealing it learns an implicit, dense reward model from demonstrations, and shows how to recover this reward to improve policies with reinforcement learning, outperforming original SFT models on benchmarks.
Conventionally, supervised fine-tuning (SFT) is treated as a simple imitation learning process that only trains a policy to imitate expert behavior on demonstration datasets. In this work, we challenge this view by establishing a fundamental equivalence between SFT and Inverse Reinforcement Learning. We prove that the SFT objective is a special case of Inverse Q-Learning, which implies that the SFT process does not just learn a policy, but also an implicit, dense, token-level reward model that explains the expert demonstrations. We then show how to recover this dense reward signal directly from the SFT model by formulating a baseline-relative reward function. The availability of such a dense reward model offers numerous benefits, providing granular credit assignment for each token generated. We demonstrate one key application by using these recovered rewards to further improve the policy with reinforcement learning. Our method, Dense-Path REINFORCE, consistently outperforms the original SFT models on instruction-following benchmarks. This work reframes SFT not merely as policy imitation but as a powerful reward learning mechanism, opening new possibilities for leveraging expert demonstrations.