An Improved Best-of-both-worlds Algorithm for Bandits with Delayed
  Feedback
Saeed Masoudian Julian Zimmert Yevgeny Seldin
We propose a new best-of-both-worlds algorithm for bandits with variably
delayed feedback. The algorithm improves on prior work by Masoudian et al.
[2022] by eliminating the need in prior knowledge of the maximal delay
$d_{\mathrm{max}}$ and providing tighter regret bounds in both regimes. The
algorithm and its regret bounds are based on counts of outstanding observations
(a quantity that is observed at action time) rather than delays or the maximal
delay (quantities that are only observed when feedback arrives). One major
contribution is a novel control of distribution drift, which is based on biased
loss estimators and skipping of observations with excessively large delays.
Another major contribution is demonstrating that the complexity of
best-of-both-worlds bandits with delayed feedback is characterized by the
cumulative count of outstanding observations after skipping of observations
with excessively large delays, rather than the delays or the maximal delay.