An Investigation of Time Reversal Symmetry in Reinforcement Learning
Brett Barkley Amy Zhang David Fridovich-Keil
One of the fundamental challenges associated with reinforcement learning (RL)
is that collecting sufficient data can be both time-consuming and expensive. In
this paper, we formalize a concept of time reversal symmetry in a Markov
decision process (MDP), which builds upon the established structure of
dynamically reversible Markov chains (DRMCs) and time-reversibility in
classical physics. Specifically, we investigate the utility of this concept in
reducing the sample complexity of reinforcement learning. We observe that
utilizing the structure of time reversal in an MDP allows every environment
transition experienced by an agent to be transformed into a feasible
reverse-time transition, effectively doubling the number of experiences in the
environment. To test the usefulness of this newly synthesized data, we develop
a novel approach called time symmetric data augmentation (TSDA) and investigate
its application in both proprioceptive and pixel-based state within the realm
of off-policy, model-free RL. Empirical evaluations showcase how these
synthetic transitions can enhance the sample efficiency of RL agents in time
reversible scenarios without friction or contact. We also test this method in
more realistic environments where these assumptions are not globally satisfied.
We find that TSDA can significantly degrade sample efficiency and policy
performance, but can also improve sample efficiency under the right conditions.
Ultimately we conclude that time symmetry shows promise in enhancing the sample
efficiency of reinforcement learning and provide guidance when the environment
and reward structures are of an appropriate form for TSDA to be employed
effectively.