Curiosity & Entropy Driven Unsupervised RL in Multiple Environments
Shaurya Dewan Anisha Jain Zoe LaLena Lifan Yu
The authors of 'Unsupervised Reinforcement Learning in Multiple environments'
propose a method, alpha-MEPOL, to tackle unsupervised RL across multiple
environments. They pre-train a task-agnostic exploration policy using
interactions from an entire environment class and then fine-tune this policy
for various tasks using supervision. We expanded upon this work, with the goal
of improving performance. We primarily propose and experiment with five new
modifications to the original work: sampling trajectories using an
entropy-based probability distribution, dynamic alpha, higher KL Divergence
threshold, curiosity-driven exploration, and alpha-percentile sampling on
curiosity. Dynamic alpha and higher KL-Divergence threshold both provided a
significant improvement over the baseline from the earlier work. PDF-sampling
failed to provide any improvement due to it being approximately equivalent to
the baseline method when the sample space is small. In high-dimensional
environments, the addition of curiosity-driven exploration enhances learning by
encouraging the agent to seek diverse experiences and explore the unknown more.
However, its benefits are limited in low-dimensional and simpler environments
where exploration possibilities are constrained and there is little that is
truly unknown to the agent. Overall, some of our experiments did boost
performance over the baseline and there are a few directions that seem
promising for further research.