Between accurate prediction and poor decision making: the AI/ML gap
Gianluca Bontempi
Intelligent agents rely on AI/ML functionalities to predict the consequence
of possible actions and optimise the policy. However, the effort of the
research community in addressing prediction accuracy has been so intense (and
successful) that it created the illusion that the more accurate the learner
prediction (or classification) the better would have been the final decision.
Now, such an assumption is valid only if the (human or artificial) decision
maker has complete knowledge of the utility of the possible actions. This paper
argues that AI/ML community has taken so far a too unbalanced approach by
devoting excessive attention to the estimation of the state (or target)
probability to the detriment of accurate and reliable estimations of the
utility. In particular, few evidence exists about the impact of a wrong utility
assessment on the resulting expected utility of the decision strategy. This
situation is creating a substantial gap between the expectations and the
effective impact of AI solutions, as witnessed by recent criticisms and
emphasised by the regulatory legislative efforts. This paper aims to study this
gap by quantifying the sensitivity of the expected utility to the utility
uncertainty and comparing it to the one due to probability estimation.
Theoretical and simulated results show that an inaccurate utility assessment
may as (and sometimes) more harmful than a poor probability estimation. The
final recommendation to the community is then to undertake a focus shift from a
pure accuracy-driven (or obsessed) approach to a more utility-aware
methodology.