Complex Claim Verification with Evidence Retrieved in the Wild
Jifan Chen Grace Kim Aniruddh Sriram Greg Durrett Eunsol Choi
Evidence retrieval is a core part of automatic fact-checking. Prior work
makes simplifying assumptions in retrieval that depart from real-world use
cases: either no access to evidence, access to evidence curated by a human
fact-checker, or access to evidence available long after the claim has been
made. In this work, we present the first fully automated pipeline to check
real-world claims by retrieving raw evidence from the web. We restrict our
retriever to only search documents available prior to the claim's making,
modeling the realistic scenario where an emerging claim needs to be checked.
Our pipeline includes five components: claim decomposition, raw document
retrieval, fine-grained evidence retrieval, claim-focused summarization, and
veracity judgment. We conduct experiments on complex political claims in the
ClaimDecomp dataset and show that the aggregated evidence produced by our
pipeline improves veracity judgments. Human evaluation finds the evidence
summary produced by our system is reliable (it does not hallucinate
information) and relevant to answering key questions about a claim, suggesting
that it can assist fact-checkers even when it cannot surface a complete
evidence set.