MRxaI: Black-Box Explainability for Image Classifiers in a Medical
  Setting
Nathan Blake Hana Chockler David A. Kelly Santiago Calderon Pena Akchunya Chanchal
Existing tools for explaining the output of image classifiers can be divided
into white-box, which rely on access to the model internals, and black-box,
agnostic to the model. As the usage of AI in the medical domain grows, so too
does the usage of explainability tools. Existing work on medical image
explanations focuses on white-box tools, such as gradcam. However, there are
clear advantages to switching to a black-box tool, including the ability to use
it with any classifier and the wide selection of black-box tools available. On
standard images, black-box tools are as precise as white-box. In this paper we
compare the performance of several black-box methods against gradcam on a brain
cancer MRI dataset. We demonstrate that most black-box tools are not suitable
for explaining medical image classifications and present a detailed analysis of
the reasons for their shortcomings. We also show that one black-box tool, a
causal explainability-based rex, performs as well as \gradcam.