Assessing Systematic Weaknesses of DNNs using Counterfactuals
Sujan Sai Gannamaneni Michael Mock Maram Akila
With the advancement of DNNs into safety-critical applications, testing
approaches for such models have gained more attention. A current direction is
the search for and identification of systematic weaknesses that put safety
assumptions based on average performance values at risk. Such weaknesses can
take on the form of (semantically coherent) subsets or areas in the input space
where a DNN performs systematically worse than its expected average. However,
it is non-trivial to attribute the reason for such observed low performances to
the specific semantic features that describe the subset. For instance,
inhomogeneities within the data w.r.t. other (non-considered) attributes might
distort results. However, taking into account all (available) attributes and
their interaction is often computationally highly expensive. Inspired by
counterfactual explanations, we propose an effective and computationally cheap
algorithm to validate the semantic attribution of existing subsets, i.e., to
check whether the identified attribute is likely to have caused the degraded
performance. We demonstrate this approach on an example from the autonomous
driving domain using highly annotated simulated data, where we show for a
semantic segmentation model that (i) performance differences among the
different pedestrian assets exist, but (ii) only in some cases is the asset
type itself the reason for this reduction in the performance.