Position: Deployed Reinforcement Learning should be Continual
For the RL community, this position paper highlights a fundamental mismatch between current deployment practices and the inherent non-stationarity of real-world environments.
The paper argues that deployed reinforcement learning agents should be continual, identifying four sources of non-stationarity that necessitate never-ending learning, and advocates moving away from the train-then-fix paradigm.
Reinforcement Learning (RL) has received increasing attention and adoption in real-world use cases. Most of these systems follow a train-then-fix paradigm, where trained agents do not learn while interacting with the world until performance degrades and retraining becomes necessary. In this position paper, we argue that deploying an agent that is incapable of optimality, but receives an evaluative reward signal, is inherently a continual RL problem. We identify four sources of non-stationarity after deployment that necessitate never-ending learning, and highlight why the best deployed agents never stop adapting. We analyze successful examples of continual RL in the real world, and present the community with the advantages and measures to move away from the current train-then-fix paradigm.