QA-NatVer: Question Answering for Natural Logic-based Fact Verification
Rami Aly Marek Strong Andreas Vlachos
Fact verification systems assess a claim's veracity based on evidence. An
important consideration in designing them is faithfulness, i.e. generating
explanations that accurately reflect the reasoning of the model. Recent works
have focused on natural logic, which operates directly on natural language by
capturing the semantic relation of spans between an aligned claim with its
evidence via set-theoretic operators. However, these approaches rely on
substantial resources for training, which are only available for high-resource
languages. To this end, we propose to use question answering to predict natural
logic operators, taking advantage of the generalization capabilities of
instruction-tuned language models. Thus, we obviate the need for annotated
training data while still relying on a deterministic inference system. In a
few-shot setting on FEVER, our approach outperforms the best baseline by $4.3$
accuracy points, including a state-of-the-art pre-trained seq2seq natural logic
system, as well as a state-of-the-art prompt-based classifier. Our system
demonstrates its robustness and portability, achieving competitive performance
on a counterfactual dataset and surpassing all approaches without further
annotation on a Danish verification dataset. A human evaluation indicates that
our approach produces more plausible proofs with fewer erroneous natural logic
operators than previous natural logic-based systems.