Binary Quantification and Dataset Shift: An Experimental Investigation
Pablo González Alejandro Moreo Fabrizio Sebastiani
Quantification is the supervised learning task that consists of training
predictors of the class prevalence values of sets of unlabelled data, and is of
special interest when the labelled data on which the predictor has been trained
and the unlabelled data are not IID, i.e., suffer from dataset shift. To date,
quantification methods have mostly been tested only on a special case of
dataset shift, i.e., prior probability shift; the relationship between
quantification and other types of dataset shift remains, by and large,
unexplored. In this work we carry out an experimental analysis of how current
quantification algorithms behave under different types of dataset shift, in
order to identify limitations of current approaches and hopefully pave the way
for the development of more broadly applicable methods. We do this by proposing
a fine-grained taxonomy of types of dataset shift, by establishing protocols
for the generation of datasets affected by these types of shift, and by testing
existing quantification methods on the datasets thus generated. One finding
that results from this investigation is that many existing quantification
methods that had been found robust to prior probability shift are not
necessarily robust to other types of dataset shift. A second finding is that no
existing quantification method seems to be robust enough to dealing with all
the types of dataset shift we simulate in our experiments. The code needed to
reproduce all our experiments is publicly available at
https://github.com/pglez82/quant_datasetshift.