We want to understand decisions, we want to shed light into the dark. The more complex an ML model’s design, the more difficult it is to describe the relationship between input and output verbally or using mathematical formulae. It is then often difficult for modellers, users, validators and also for us supervisors to verify the results in detail.
And every stakeholder also has a different understanding of and need for explainability. Checking that a model’s behaviour overall is explainable and plausible is therefore more important than for every detail to be verifiable. Is there a logical relationship between input and output? Does the model behave as we would expect where decisions are verifiable? In short, does it make sense?