Think Before You Duel: Understanding Complexities of Preference Learning
  under Constrained Resources
Rohan Deb Aadirupa Saha
We consider the problem of reward maximization in the dueling bandit setup
along with constraints on resource consumption. As in the classic dueling
bandits, at each round the learner has to choose a pair of items from a set of
$K$ items and observe a relative feedback for the current pair. Additionally,
for both items, the learner also observes a vector of resource consumptions.
The objective of the learner is to maximize the cumulative reward, while
ensuring that the total consumption of any resource is within the allocated
budget. We show that due to the relative nature of the feedback, the problem is
more difficult than its bandit counterpart and that without further assumptions
the problem is not learnable from a regret minimization perspective.
Thereafter, by exploiting assumptions on the available budget, we provide an
EXP3 based dueling algorithm that also considers the associated consumptions
and show that it achieves an
$\tilde{\mathcal{O}}\left({\frac{OPT^{(b)}}{B}}K^{1/3}T^{2/3}\right)$ regret,
where $OPT^{(b)}$ is the optimal value and $B$ is the available budget.
Finally, we provide numerical simulations to demonstrate the efficacy of our
proposed method.