Comparing Knowledge Sources for Open-Domain Scientific Claim
  Verification
Juraj Vladika Florian Matthes
The increasing rate at which scientific knowledge is discovered and health
claims shared online has highlighted the importance of developing efficient
fact-checking systems for scientific claims. The usual setting for this task in
the literature assumes that the documents containing the evidence for claims
are already provided and annotated or contained in a limited corpus. This
renders the systems unrealistic for real-world settings where knowledge sources
with potentially millions of documents need to be queried to find relevant
evidence. In this paper, we perform an array of experiments to test the
performance of open-domain claim verification systems. We test the final
verdict prediction of systems on four datasets of biomedical and health claims
in different settings. While keeping the pipeline's evidence selection and
verdict prediction parts constant, document retrieval is performed over three
common knowledge sources (PubMed, Wikipedia, Google) and using two different
information retrieval techniques. We show that PubMed works better with
specialized biomedical claims, while Wikipedia is more suited for everyday
health concerns. Likewise, BM25 excels in retrieval precision, while semantic
search in recall of relevant evidence. We discuss the results, outline frequent
retrieval patterns and challenges, and provide promising future directions.