Online Learning with Costly Features in Non-stationary Environments
Saeed Ghoorchian Evgenii Kortukov Setareh Maghsudi
Maximizing long-term rewards is the primary goal in sequential
decision-making problems. The majority of existing methods assume that side
information is freely available, enabling the learning agent to observe all
features' states before making a decision. In real-world problems, however,
collecting beneficial information is often costly. That implies that, besides
individual arms' reward, learning the observations of the features' states is
essential to improve the decision-making strategy. The problem is aggravated in
a non-stationary environment where reward and cost distributions undergo abrupt
changes over time. To address the aforementioned dual learning problem, we
extend the contextual bandit setting and allow the agent to observe subsets of
features' states. The objective is to maximize the long-term average gain,
which is the difference between the accumulated rewards and the paid costs on
average. Therefore, the agent faces a trade-off between minimizing the cost of
information acquisition and possibly improving the decision-making process
using the obtained information. To this end, we develop an algorithm that
guarantees a sublinear regret in time. Numerical results demonstrate the
superiority of our proposed policy in a real-world scenario.