LinkTransformer: A Unified Package for Record Linkage with Transformer
  Language Models
Abhishek Arora Melissa Dell
Linking information across sources is fundamental to a variety of analyses in
social science, business, and government. While large language models (LLMs)
offer enormous promise for improving record linkage in noisy datasets, in many
domains approximate string matching packages in popular softwares such as R and
Stata remain predominant. These packages have clean, simple interfaces and can
be easily extended to a diversity of languages. Our open-source package
LinkTransformer aims to extend the familiarity and ease-of-use of popular
string matching methods to deep learning. It is a general purpose package for
record linkage with transformer LLMs that treats record linkage as a text
retrieval problem. At its core is an off-the-shelf toolkit for applying
transformer models to record linkage with four lines of code. LinkTransformer
contains a rich repository of pre-trained transformer semantic similarity
models for multiple languages and supports easy integration of any transformer
language model from Hugging Face or OpenAI. It supports standard functionality
such as blocking and linking on multiple noisy fields. LinkTransformer APIs
also perform other common text data processing tasks, e.g., aggregation, noisy
de-duplication, and translation-free cross-lingual linkage. Importantly,
LinkTransformer also contains comprehensive tools for efficient model tuning,
to facilitate different levels of customization when off-the-shelf models do
not provide the required accuracy. Finally, to promote reusability,
reproducibility, and extensibility, LinkTransformer makes it easy for users to
contribute their custom-trained models to its model hub. By combining
transformer language models with intuitive APIs that will be familiar to many
users of popular string matching packages, LinkTransformer aims to democratize
the benefits of LLMs among those who may be less familiar with deep learning
frameworks.