Generating the Ground Truth: Synthetic Data for Label Noise Research
Sjoerd de Vries Dirk Thierens
Most real-world classification tasks suffer from label noise to some extent.
Such noise in the data adversely affects the generalization error of learned
models and complicates the evaluation of noise-handling methods, as their
performance cannot be accurately measured without clean labels. In label noise
research, typically either noisy or incomplex simulated data are accepted as a
baseline, into which additional noise with known properties is injected. In
this paper, we propose SYNLABEL, a framework that aims to improve upon the
aforementioned methodologies. It allows for creating a noiseless dataset
informed by real data, by either pre-specifying or learning a function and
defining it as the ground truth function from which labels are generated.
Furthermore, by resampling a number of values for selected features in the
function domain, evaluating the function and aggregating the resulting labels,
each data point can be assigned a soft label or label distribution. Such
distributions allow for direct injection and quantification of label noise. The
generated datasets serve as a clean baseline of adjustable complexity into
which different types of noise may be introduced. We illustrate how the
framework can be applied, how it enables quantification of label noise and how
it improves over existing methodologies.