Simple Modification of the Upper Confidence Bound Algorithm by
  Generalized Weighted Averages
Nobuhito Manome Shuji Shinohara Ung-il Chung
The multi-armed bandit (MAB) problem is a classical problem that models
sequential decision-making under uncertainty in reinforcement learning. In this
study, we propose a new generalized upper confidence bound (UCB) algorithm
(GWA-UCB1) by extending UCB1, which is a representative algorithm for MAB
problems, using generalized weighted averages, and present an effective
algorithm for various problem settings. GWA-UCB1 is a two-parameter
generalization of the balance between exploration and exploitation in UCB1 and
can be implemented with a simple modification of the UCB1 formula. Therefore,
this algorithm can be easily applied to UCB-based reinforcement learning
models. In preliminary experiments, we investigated the optimal parameters of a
simple generalized UCB1 (G-UCB1), prepared for comparison and GWA-UCB1, in a
stochastic MAB problem with two arms. Subsequently, we confirmed the
performance of the algorithms with the investigated parameters on stochastic
MAB problems when arm reward probabilities were sampled from uniform or normal
distributions and on survival MAB problems assuming more realistic situations.
GWA-UCB1 outperformed G-UCB1, UCB1-Tuned, and Thompson sampling in most problem
settings and can be useful in many situations. The code is available at
https://github.com/manome/python-mab.