Improving Opinion-based Question Answering Systems Through Label Error
  Detection and Overwrite
Xiao Yang Ahmed K. Mohamed Shashank Jain Stanislav Peshterliev Debojeet Chatterjee Hanwen Zha Nikita Bhalla Gagan Aneja Pranab Mohanty
Label error is a ubiquitous problem in annotated data. Large amounts of label
error substantially degrades the quality of deep learning models. Existing
methods to tackle the label error problem largely focus on the classification
task, and either rely on task specific architecture or require non-trivial
additional computations, which is undesirable or even unattainable for industry
usage. In this paper, we propose LEDO: a model-agnostic and computationally
efficient framework for Label Error Detection and Overwrite. LEDO is based on
Monte Carlo Dropout combined with uncertainty metrics, and can be easily
generalized to multiple tasks and data sets. Applying LEDO to an industry
opinion-based question answering system demonstrates it is effective at
improving accuracy in all the core models. Specifically, LEDO brings 1.1% MRR
gain for the retrieval model, 1.5% PR AUC improvement for the machine reading
comprehension model, and 0.9% rise in the Average Precision for the ranker, on
top of the strong baselines with a large-scale social media dataset.
Importantly, LEDO is computationally efficient compared to methods that require
loss function change, and cost-effective as the resulting data can be used in
the same continuous training pipeline for production. Further analysis shows
that these gains come from an improved decision boundary after cleaning the
label errors existed in the training data.