Enumerating the k-fold configurations in multi-class classification
  problems
Attila Fazekas Gyorgy Kovacs
K-fold cross-validation is a widely used tool for assessing classifier
performance. The reproducibility crisis faced by artificial intelligence partly
results from the irreproducibility of reported k-fold cross-validation-based
performance scores. Recently, we introduced numerical techniques to test the
consistency of claimed performance scores and experimental setups. In a crucial
use case, the method relies on the combinatorial enumeration of all k-fold
configurations, for which we proposed an algorithm in the binary classification
case.