Reinforcement Learning with Fast and Forgetful Memory
Steven Morad Ryan Kortvelesy Stephan Liwicki Amanda Prorok
Nearly all real world tasks are inherently partially observable,
necessitating the use of memory in Reinforcement Learning (RL). Most model-free
approaches summarize the trajectory into a latent Markov state using memory
models borrowed from Supervised Learning (SL), even though RL tends to exhibit
different training and efficiency characteristics. Addressing this discrepancy,
we introduce Fast and Forgetful Memory, an algorithm-agnostic memory model
designed specifically for RL. Our approach constrains the model search space
via strong structural priors inspired by computational psychology. It is a
drop-in replacement for recurrent neural networks (RNNs) in recurrent RL
algorithms, achieving greater reward than RNNs across various recurrent
benchmarks and algorithms without changing any hyperparameters. Moreover, Fast
and Forgetful Memory exhibits training speeds two orders of magnitude faster
than RNNs, attributed to its logarithmic time and linear space complexity. Our
implementation is available at https://github.com/proroklab/ffm.