Leveraging heterogeneous spillover effects in maximizing contextual
  bandit rewards
Ahmed Sayeed Faruk Elena Zheleva
Recommender systems relying on contextual multi-armed bandits continuously
improve relevant item recommendations by taking into account the contextual
information. The objective of these bandit algorithms is to learn the best arm
(i.e., best item to recommend) for each user and thus maximize the cumulative
rewards from user engagement with the recommendations. However, current
approaches ignore potential spillover between interacting users, where the
action of one user can impact the actions and rewards of other users. Moreover,
spillover may vary for different people based on their preferences and the
closeness of ties to other users. This leads to heterogeneity in the spillover
effects, i.e., the extent to which the action of one user can impact the action
of another. Here, we propose a framework that allows contextual multi-armed
bandits to account for such heterogeneous spillovers when choosing the best arm
for each user. By experimenting on several real-world datasets using prominent
linear and non-linear contextual bandit algorithms, we observe that our
proposed method leads to significantly higher rewards than existing solutions
that ignore spillover.