Evaluation of Popular XAI Applied to Clinical Prediction Models: Can
  They be Trusted?
Aida Brankovic David Cook Jessica Rahman Wenjie Huang Sankalp Khanna
The absence of transparency and explainability hinders the clinical adoption
of Machine learning (ML) algorithms. Although various methods of explainable
artificial intelligence (XAI) have been suggested, there is a lack of
literature that delves into their practicality and assesses them based on
criteria that could foster trust in clinical environments. To address this gap
this study evaluates two popular XAI methods used for explaining predictive
models in the healthcare context in terms of whether they (i) generate
domain-appropriate representation, i.e. coherent with respect to the
application task, (ii) impact clinical workflow and (iii) are consistent. To
that end, explanations generated at the cohort and patient levels were
analysed. The paper reports the first benchmarking of the XAI methods applied
to risk prediction models obtained by evaluating the concordance between
generated explanations and the trigger of a future clinical deterioration
episode recorded by the data collection system. We carried out an analysis
using two Electronic Medical Records (EMR) datasets sourced from Australian
major hospitals. The findings underscore the limitations of state-of-the-art
XAI methods in the clinical context and their potential benefits. We discuss
these limitations and contribute to the theoretical development of trustworthy
XAI solutions where clinical decision support guides the choice of intervention
by suggesting the pattern or drivers for clinical deterioration in the future.