Data-Driven Estimation of the False Positive Rate of the Bayes Binary
  Classifier via Soft Labels
Minoh Jeong Martina Cardone Alex Dytso
Classification is a fundamental task in many applications on which
data-driven methods have shown outstanding performances. However, it is
challenging to determine whether such methods have achieved the optimal
performance. This is mainly because the best achievable performance is
typically unknown and hence, effectively estimating it is of prime importance.
In this paper, we consider binary classification problems and we propose an
estimator for the false positive rate (FPR) of the Bayes classifier, that is,
the optimal classifier with respect to accuracy, from a given dataset. Our
method utilizes soft labels, or real-valued labels, which are gaining
significant traction thanks to their properties. We thoroughly examine various
theoretical properties of our estimator, including its consistency,
unbiasedness, rate of convergence, and variance. To enhance the versatility of
our estimator beyond soft labels, we also consider noisy labels, which
encompass binary labels. For noisy labels, we develop effective FPR estimators
by leveraging a denoising technique and the Nadaraya-Watson estimator. Due to
the symmetry of the problem, our results can be readily applied to estimate the
false negative rate of the Bayes classifier.