Intrinsic Data Constraints and Upper Bounds in Binary Classification
  Performance
Fei Jing Zi-Ke Zhang Qingpeng Zhang
The structure of data organization is widely recognized as having a
substantial influence on the efficacy of machine learning algorithms,
particularly in binary classification tasks. Our research provides a
theoretical framework suggesting that the maximum potential of binary
classifiers on a given dataset is primarily constrained by the inherent
qualities of the data. Through both theoretical reasoning and empirical
examination, we employed standard objective functions, evaluative metrics, and
binary classifiers to arrive at two principal conclusions. Firstly, we show
that the theoretical upper bound of binary classification performance on actual
datasets can be theoretically attained. This upper boundary represents a
calculable equilibrium between the learning loss and the metric of evaluation.
Secondly, we have computed the precise upper bounds for three commonly used
evaluation metrics, uncovering a fundamental uniformity with our overarching
thesis: the upper bound is intricately linked to the dataset's characteristics,
independent of the classifier in use. Additionally, our subsequent analysis
uncovers a detailed relationship between the upper limit of performance and the
level of class overlap within the binary classification data. This relationship
is instrumental for pinpointing the most effective feature subsets for use in
feature engineering.