How to Evaluate Entity Resolution Systems: An Entity-Centric Framework
  with Application to Inventor Name Disambiguation
Olivier Binette Youngsoo Baek Siddharth Engineer Christina Jones Abel Dasylva Jerome P. Reiter
Entity resolution (record linkage, microclustering) systems are notoriously
difficult to evaluate. Looking for a needle in a haystack, traditional
evaluation methods use sophisticated, application-specific sampling schemes to
find matching pairs of records among an immense number of non-matches. We
propose an alternative that facilitates the creation of representative,
reusable benchmark data sets without necessitating complex sampling schemes.
These benchmark data sets can then be used for model training and a variety of
evaluation tasks. Specifically, we propose an entity-centric data labeling
methodology that integrates with a unified framework for monitoring summary
statistics, estimating key performance metrics such as cluster and pairwise
precision and recall, and analyzing root causes for errors. We validate the
framework in an application to inventor name disambiguation and through
simulation studies. Software: https://github.com/OlivierBinette/er-evaluation/