Reinforcement Learning with an Abrupt Model Change
This work addresses a specific challenge in reinforcement learning for scenarios where models change abruptly, but it appears incremental as it builds on existing change detection methods.
The paper tackles the problem of reinforcement learning in environments that undergo abrupt model changes, proposing a model-free algorithm that learns the optimal policy by interacting with the environment and using quickest change detection to handle changes, with effectiveness demonstrated through simulation results.
The problem of reinforcement learning is considered where the environment or the model undergoes a change. An algorithm is proposed that an agent can apply in such a problem to achieve the optimal long-time discounted reward. The algorithm is model-free and learns the optimal policy by interacting with the environment. It is shown that the proposed algorithm has strong optimality properties. The effectiveness of the algorithm is also demonstrated using simulation results. The proposed algorithm exploits a fundamental reward-detection trade-off present in these problems and uses a quickest change detection algorithm to detect the model change. Recommendations are provided for faster detection of model changes and for smart initialization strategies.