On Reinforcement Learning, Effect Handlers, and the State Monad
This work provides a modular approach for embedding reinforcement learning in functional languages, which is incremental as it builds on existing algebraic effects frameworks.
The authors tackled the problem of integrating reinforcement learning into functional programming by using algebraic effects and handlers to separate the learning algorithm from its environment, achieving modularity and clear separation of concerns.
We study the algebraic effects and handlers as a way to support decision-making abstractions in functional programs, whereas a user can ask a learning algorithm to resolve choices without implementing the underlying selection mechanism, and give a feedback by way of rewards. Differently from some recently proposed approach to the problem based on the selection monad [Abadi and Plotkin, LICS 2021], we express the underlying intelligence as a reinforcement learning algorithm implemented as a set of handlers for some of these algebraic operations, including those for choices and rewards. We show how we can in practice use algebraic operations and handlers -- as available in the programming language EFF -- to clearly separate the learning algorithm from its environment, thus allowing for a good level of modularity. We then show how the host language can be taken as a lambda-calculus with handlers, this way showing what the essential linguistic features are. We conclude by hinting at how type and effect systems could ensure safety properties, at the same time pointing at some directions for further work.