User-Interactive Offline Reinforcement Learning
This addresses the issue of practical trust and control for users of offline RL systems, though it is incremental as it builds on existing methods by adding interactive tuning.
The paper tackles the problem of trust and hyperparameter tuning in offline reinforcement learning by proposing an algorithm that allows users to interactively adjust the proximity of the learned policy to the original policy at runtime, enabling them to start with familiar behavior and incrementally grant deviations while monitoring for deterioration.
Offline reinforcement learning algorithms still lack trust in practice due to the risk that the learned policy performs worse than the original policy that generated the dataset or behaves in an unexpected way that is unfamiliar to the user. At the same time, offline RL algorithms are not able to tune their most important hyperparameter - the proximity of the learned policy to the original policy. We propose an algorithm that allows the user to tune this hyperparameter at runtime, thereby addressing both of the above mentioned issues simultaneously. This allows users to start with the original behavior and grant successively greater deviation, as well as stopping at any time when the policy deteriorates or the behavior is too far from the familiar one.