Distributed Record Linkage in Healthcare Data with Apache Spark
Mohammad Heydari Reza Sarshar Mohammad Ali Soltanshahi
Healthcare data is a valuable resource for research, analysis, and
decision-making in the medical field. However, healthcare data is often
fragmented and distributed across various sources, making it challenging to
combine and analyze effectively. Record linkage, also known as data matching,
is a crucial step in integrating and cleaning healthcare data to ensure data
quality and accuracy. Apache Spark, a powerful open-source distributed big data
processing framework, provides a robust platform for performing record linkage
tasks with the aid of its machine learning library. In this study, we developed
a new distributed data-matching model based on the Apache Spark Machine
Learning library. To ensure the correct functioning of our model, the
validation phase has been performed on the training data. The main challenge is
data imbalance because a large amount of data is labeled false, and a small
number of records are labeled true. By utilizing SVM and Regression algorithms,
our results demonstrate that research data was neither over-fitted nor
under-fitted, and this shows that our distributed model works well on the data.