Generalized Population-Based Training for Hyperparameter Optimization in
  Reinforcement Learning
Hui Bai Ran Cheng
Hyperparameter optimization plays a key role in the machine learning domain.
Its significance is especially pronounced in reinforcement learning (RL), where
agents continuously interact with and adapt to their environments, requiring
dynamic adjustments in their learning trajectories. To cater to this
dynamicity, the Population-Based Training (PBT) was introduced, leveraging the
collective intelligence of a population of agents learning simultaneously.
However, PBT tends to favor high-performing agents, potentially neglecting the
explorative potential of agents on the brink of significant advancements. To
mitigate the limitations of PBT, we present the Generalized Population-Based
Training (GPBT), a refined framework designed for enhanced granularity and
flexibility in hyperparameter adaptation. Complementing GPBT, we further
introduce Pairwise Learning (PL). Instead of merely focusing on elite agents,
PL employs a comprehensive pairwise strategy to identify performance
differentials and provide holistic guidance to underperforming agents. By
integrating the capabilities of GPBT and PL, our approach significantly
improves upon traditional PBT in terms of adaptability and computational
efficiency. Rigorous empirical evaluations across a range of RL benchmarks
confirm that our approach consistently outperforms not only the conventional
PBT but also its Bayesian-optimized variant.