Capsa: A Unified Framework for Quantifying Risk in Deep Neural Networks
Sadhana Lolla Iaroslav Elistratov Alejandro Perez Elaheh Ahmadi Daniela Rus Alexander Amini
The modern pervasiveness of large-scale deep neural networks (NNs) is driven
by their extraordinary performance on complex problems but is also plagued by
their sudden, unexpected, and often catastrophic failures, particularly on
challenging scenarios. Existing algorithms that provide risk-awareness to NNs
are complex and ad-hoc. Specifically, these methods require significant
engineering changes, are often developed only for particular settings, and are
not easily composable. Here we present capsa, a framework for extending models
with risk-awareness. Capsa provides a methodology for quantifying multiple
forms of risk and composing different algorithms together to quantify different
risk metrics in parallel. We validate capsa by implementing state-of-the-art
uncertainty estimation algorithms within the capsa framework and benchmarking
them on complex perception datasets. We demonstrate capsa's ability to easily
compose aleatoric uncertainty, epistemic uncertainty, and bias estimation
together in a single procedure, and show how this approach provides a
comprehensive awareness of NN risk.