A Lightweight Measure of Classification Difficulty from Application
  Dataset Characteristics
Bryan Bo Cao Abhinav Sharma Lawrence O'Gorman Michael Coss Shubham Jain
Despite accuracy and computation benchmarks being widely available to help
choose among neural network models, these are usually trained on datasets with
many classes, and do not give a precise idea of performance for applications of
few (< 10) classes. The conventional procedure to predict performance is to
train and test repeatedly on the different models and dataset variations of
interest. However, this is computationally expensive. We propose an efficient
classification difficulty measure that is calculated from the number of classes
and intra- and inter-class similarity metrics of the dataset. After a single
stage of training and testing per model family, relative performance for
different datasets and models of the same family can be predicted by comparing
difficulty measures - without further training and testing. We show how this
measure can help a practitioner select a computationally efficient model for a
small dataset 6 to 29x faster than through repeated training and testing. We
give an example of use of the measure for an industrial application in which
options are identified to select a model 42% smaller than the baseline
YOLOv5-nano model, and if class merging from 3 to 2 classes meets requirements,
85% smaller.