On the Actionability of Outcome Prediction
Lydia T. Liu Solon Barocas Jon Kleinberg Karen Levy
Predicting future outcomes is a prevalent application of machine learning in
social impact domains. Examples range from predicting student success in
education to predicting disease risk in healthcare. Practitioners recognize
that the ultimate goal is not just to predict but to act effectively.
Increasing evidence suggests that relying on outcome predictions for downstream
interventions may not have desired results.
  In most domains there exists a multitude of possible interventions for each
individual, making the challenge of taking effective action more acute. Even
when causal mechanisms connecting the individual's latent states to outcomes is
well understood, in any given instance (a specific student or patient),
practitioners still need to infer -- from budgeted measurements of latent
states -- which of many possible interventions will be most effective for this
individual. With this in mind, we ask: when are accurate predictors of outcomes
helpful for identifying the most suitable intervention?
  Through a simple model encompassing actions, latent states, and measurements,
we demonstrate that pure outcome prediction rarely results in the most
effective policy for taking actions, even when combined with other
measurements. We find that except in cases where there is a single decisive
action for improving the outcome, outcome prediction never maximizes "action
value", the utility of taking actions. Making measurements of actionable latent
states, where specific actions lead to desired outcomes, considerably enhances
the action value compared to outcome prediction, and the degree of improvement
depends on action costs and the outcome model. This analysis emphasizes the
need to go beyond generic outcome prediction in interventional settings by
incorporating knowledge of plausible actions and latent states.