Employing Label Models on ChatGPT Answers Improves Legal Text Entailment
  Performance
Chau Nguyen Le-Minh Nguyen
The objective of legal text entailment is to ascertain whether the assertions
in a legal query logically follow from the information provided in one or
multiple legal articles. ChatGPT, a large language model, is robust in many
natural language processing tasks, including legal text entailment: when we set
the temperature = 0 (the ChatGPT answers are deterministic) and prompt the
model, it achieves 70.64% accuracy on COLIEE 2022 dataset, which outperforms
the previous SOTA of 67.89%. On the other hand, if the temperature is larger
than zero, ChatGPT answers are not deterministic, leading to inconsistent
answers and fluctuating results. We propose to leverage label models (a
fundamental component of weak supervision techniques) to integrate the
provisional answers by ChatGPT into consolidated labels. By that way, we treat
ChatGPT provisional answers as noisy predictions which can be consolidated by
label models. The experimental results demonstrate that this approach can
attain an accuracy of 76.15%, marking a significant improvement of 8.26% over
the prior state-of-the-art benchmark. Additionally, we perform an analysis of
the instances where ChatGPT produces incorrect answers, then we classify the
errors, offering insights that could guide potential enhancements for future
research endeavors.