Personalized Reinforcement Learning with a Budget of Policies
Dmitry Ivanov Omer Ben-Porat
Personalization in machine learning (ML) tailors models' decisions to the
individual characteristics of users. While this approach has seen success in
areas like recommender systems, its expansion into high-stakes fields such as
healthcare and autonomous driving is hindered by the extensive regulatory
approval processes involved. To address this challenge, we propose a novel
framework termed represented Markov Decision Processes (r-MDPs) that is
designed to balance the need for personalization with the regulatory
constraints. In an r-MDP, we cater to a diverse user population, each with
unique preferences, through interaction with a small set of representative
policies. Our objective is twofold: efficiently match each user to an
appropriate representative policy and simultaneously optimize these policies to
maximize overall social welfare. We develop two deep reinforcement learning
algorithms that efficiently solve r-MDPs. These algorithms draw inspiration
from the principles of classic K-means clustering and are underpinned by robust
theoretical foundations. Our empirical investigations, conducted across a
variety of simulated environments, showcase the algorithms' ability to
facilitate meaningful personalization even under constrained policy budgets.
Furthermore, they demonstrate scalability, efficiently adapting to larger
policy budgets.