Performative Reinforcement Learning in Gradually Shifting Environments
Ben Rank Stelios Triantafyllou Debmalya Mandal Goran Radanovic
When Reinforcement Learning (RL) agents are deployed in practice, they might
impact their environment and change its dynamics. Ongoing research attempts to
formally model this phenomenon and to analyze learning algorithms in these
models. To this end, we propose a framework where the current environment
depends on the deployed policy as well as its previous dynamics. This is a
generalization of Performative RL (PRL) [Mandal et al., 2023]. Unlike PRL, our
framework allows to model scenarios where the environment gradually adjusts to
a deployed policy. We adapt two algorithms from the performative prediction
literature to our setting and propose a novel algorithm called Mixed Delayed
Repeated Retraining (MDRR). We provide conditions under which these algorithms
converge and compare them using three metrics: number of retrainings,
approximation guarantee, and number of samples per deployment. Unlike previous
approaches, MDRR combines samples from multiple deployments in its training.
This makes MDRR particularly suitable for scenarios where the environment's
response strongly depends on its previous dynamics, which are common in
practice. We experimentally compare the algorithms using a simulation-based
testbed and our results show that MDRR converges significantly faster than
previous approaches.