Personalized Reinforcement Learning with a Budget of Policies
This work addresses the problem of enabling personalization in high-stakes fields like healthcare and autonomous driving for developers and regulators, though it appears incremental as it builds on existing RL and clustering methods.
The paper tackles the challenge of personalizing reinforcement learning for diverse user populations under regulatory constraints by introducing represented Markov Decision Processes (r-MDPs), which match users to a small set of representative policies and optimize them for social welfare, achieving scalability and meaningful personalization in simulated environments.
Personalization in machine learning (ML) tailors models' decisions to the individual characteristics of users. While this approach has seen success in areas like recommender systems, its expansion into high-stakes fields such as healthcare and autonomous driving is hindered by the extensive regulatory approval processes involved. To address this challenge, we propose a novel framework termed represented Markov Decision Processes (r-MDPs) that is designed to balance the need for personalization with the regulatory constraints. In an r-MDP, we cater to a diverse user population, each with unique preferences, through interaction with a small set of representative policies. Our objective is twofold: efficiently match each user to an appropriate representative policy and simultaneously optimize these policies to maximize overall social welfare. We develop two deep reinforcement learning algorithms that efficiently solve r-MDPs. These algorithms draw inspiration from the principles of classic K-means clustering and are underpinned by robust theoretical foundations. Our empirical investigations, conducted across a variety of simulated environments, showcase the algorithms' ability to facilitate meaningful personalization even under constrained policy budgets. Furthermore, they demonstrate scalability, efficiently adapting to larger policy budgets.