Learning Kernel-Based MDPs from Episodic Preferential Feedback
It establishes theoretical foundations for RLHF in a general kernel MDP setting, addressing a key gap in understanding preference-only learning.
This paper provides the first rigorous regret analysis for preference-based reinforcement learning in episodic kernel MDPs, achieving sublinear regret bounds that guarantee convergence to the optimal policy.
Human feedback often arrives as preferences rather than calibrated numeric rewards, motivating reinforcement learning from preferential feedback, also referred to as reinforcement learning from human feedback (RLHF). We present a rigorous theoretical study of preference-only learning in episodic kernel MDPs. In each episode, the learner deploys two policies from a common start state and receives a single binary label indicating which trajectory is preferred, modeled by a Bradley--Terry--Luce link on the difference of cumulative (unobserved) rewards. Under kernel-based assumptions on the reward and transition functions (one of the most general models amenable to theoretical analysis) we develop preference-based value estimation and confidence sets tailored to end-of-episode comparisons.We prove high-probability regret bounds that scale sublinearly in the number of episodes, implying that the value of the learned policy converges to that of the optimal policy.