And every stakeholder also has a different understanding of and need for explainability. Checking that a model’s behaviour overall is explainable and plausible is therefore more important than for every detail to be verifiable. Is there a logical relationship between input and output? Does the model behave as we would expect where decisions are verifiable? In short, does it make sense?
It will often be a matter of treading a golden middle way of the “grey box”, between the need for explanation on the one hand and performance on the other. What is important is understanding and critically assessing the methodology in question; it is not a question of every single process having to be signed off by a human being.