Is AUC the best measure for practical comparison of anomaly detectors?
Vít Škvára Tomáš Pevný Václav Šmídl
The area under receiver operating characteristics (AUC) is the standard
measure for comparison of anomaly detectors. Its advantage is in providing a
scalar number that allows a natural ordering and is independent on a threshold,
which allows to postpone the choice. In this work, we question whether AUC is a
good metric for anomaly detection, or if it gives a false sense of comfort, due
to relying on assumptions which are unlikely to hold in practice. Our
investigation shows that variations of AUC emphasizing accuracy at low false
positive rate seem to be better correlated with the needs of practitioners, but
also that we can compare anomaly detectors only in the case when we have
representative examples of anomalous samples. This last result is disturbing,
as it suggests that in many cases, we should do active or few-show learning
instead of pure anomaly detection.