Limited Resource Allocation in a Non-Markovian World: The Case of
  Maternal and Child Healthcare
Panayiotis Danassis Shresth Verma Jackson A. Killian Aparna Taneja Milind Tambe
The success of many healthcare programs depends on participants' adherence.
We consider the problem of scheduling interventions in low resource settings
(e.g., placing timely support calls from health workers) to increase adherence
and/or engagement. Past works have successfully developed several classes of
Restless Multi-armed Bandit (RMAB) based solutions for this problem.
Nevertheless, all past RMAB approaches assume that the participants' behaviour
follows the Markov property. We demonstrate significant deviations from the
Markov assumption on real-world data on a maternal health awareness program
from our partner NGO, ARMMAN. Moreover, we extend RMABs to continuous state
spaces, a previously understudied area. To tackle the generalised non-Markovian
RMAB setting we (i) model each participant's trajectory as a time-series, (ii)
leverage the power of time-series forecasting models to learn complex patterns
and dynamics to predict future states, and (iii) propose the Time-series Arm
Ranking Index (TARI) policy, a novel algorithm that selects the RMAB arms that
will benefit the most from an intervention, given our future state predictions.
We evaluate our approach on both synthetic data, and a secondary analysis on
real data from ARMMAN, and demonstrate significant increase in engagement
compared to the SOTA, deployed Whittle index solution. This translates to 16.3
hours of additional content listened, 90.8% more engagement drops prevented,
and reaching more than twice as many high dropout-risk beneficiaries.