Optimistic Transfer under Task Shift via Bellman Alignment
This work addresses the challenge of efficient knowledge transfer in reinforcement learning for scenarios with related tasks, offering a model-agnostic solution that is incremental but provides specific gains.
The paper tackles the problem of online transfer reinforcement learning where naive reuse of source task data introduces bias, by proposing a Bellman alignment method that corrects for task mismatch. The result is a two-stage Q-learning framework that achieves regret bounds scaling with task shift complexity and shows empirical improvements in tabular and neural network settings.
We study online transfer reinforcement learning (RL) in episodic Markov decision processes, where experience from related source tasks is available during learning on a target task. A fundamental difficulty is that task similarity is typically defined in terms of rewards or transitions, whereas online RL algorithms operate on Bellman regression targets. As a result, naively reusing source Bellman updates introduces systematic bias and invalidates regret guarantees. We identify one-step Bellman alignment as the correct abstraction for transfer in online RL and propose re-weighted targeting (RWT), an operator-level correction that retargets continuation values and compensates for transition mismatch via a change of measure. RWT reduces task mismatch to a fixed one-step correction and enables statistically sound reuse of source data. This alignment yields a two-stage RWT $Q$-learning framework that separates variance reduction from bias correction. Under RKHS function approximation, we establish regret bounds that scale with the complexity of the task shift rather than the target MDP. Empirical results in both tabular and neural network settings demonstrate consistent improvements over single-task learning and naïve pooling, highlighting Bellman alignment as a model-agnostic transfer principle for online RL.