Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Data
This addresses the challenge of reducing engineering costs in RL applications where offline data is available but additional online data is needed, offering a solution for scenarios requiring efficient policy finetuning.
The paper tackles the problem of improving reinforcement learning policies using offline data and limited online exploration, proposing an algorithm with provable guarantees to design a single non-reactive exploration policy. The result is analyzed theoretically, measuring final policy quality based on local coverage of the offline dataset and additional data collected.
In some applications of reinforcement learning, a dataset of pre-collected experience is already available but it is also possible to acquire some additional online data to help improve the quality of the policy. However, it may be preferable to gather additional data with a single, non-reactive exploration policy and avoid the engineering costs associated with switching policies. In this paper we propose an algorithm with provable guarantees that can leverage an offline dataset to design a single non-reactive policy for exploration. We theoretically analyze the algorithm and measure the quality of the final policy as a function of the local coverage of the original dataset and the amount of additional data collected.