SMX: Sequential Monte Carlo Planning for Expert Iteration
Matthew V Macfarlane Edan Toledo Donal Byrne Siddarth Singh Paul Duckworth Alexandre Laterre
Developing agents that can leverage planning abilities during their decision
and learning processes is critical to the advancement of Artificial
Intelligence. Recent works have demonstrated the effectiveness of combining
tree-based search methods and self-play learning mechanisms. Yet, these methods
typically face scaling challenges due to the sequential nature of their search.
While practical engineering solutions can partly overcome this, they still
demand extensive computational resources, which hinders their applicability. In
this paper, we introduce SMX, a model-based planning algorithm that utilises
scalable Sequential Monte Carlo methods to create an effective self-learning
mechanism. Grounded in the theoretical framework of control as inference, SMX
benefits from robust theoretical underpinnings. Its sampling-based search
approach makes it adaptable to environments with both discrete and continuous
action spaces. Furthermore, SMX allows for high parallelisation and can run on
hardware accelerators to optimise computing efficiency. SMX demonstrates a
statistically significant improvement in performance compared to AlphaZero, as
well as demonstrating its performance as an improvement operator for a
model-free policy, matching or exceeding top model-free methods across both
continuous and discrete environments.