TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions
Jamshid Mozafari Anubhav Jangra Adam Jatowt
Nowadays, individuals tend to engage in dialogues with Large Language Models,
seeking answers to their questions. In times when such answers are readily
accessible to anyone, the stimulation and preservation of human's cognitive
abilities, as well as the assurance of maintaining good reasoning skills by
humans becomes crucial. This study addresses such needs by proposing hints
(instead of final answers or before giving answers) as a viable solution. We
introduce a framework for the automatic hint generation for factoid questions,
employing it to construct TriviaHG, a novel large-scale dataset featuring
160,230 hints corresponding to 16,645 questions from the TriviaQA dataset.
Additionally, we present an automatic evaluation method that measures the
Convergence and Familiarity quality attributes of hints. To evaluate the
TriviaHG dataset and the proposed evaluation method, we enlisted 10 individuals
to annotate 2,791 hints and tasked 6 humans with answering questions using the
provided hints. The effectiveness of hints varied, with success rates of 96%,
78%, and 36% for questions with easy, medium, and hard answers, respectively.
Moreover, the proposed automatic evaluation methods showed a robust correlation
with annotators' results. Conclusively, the findings highlight three key
insights: the facilitative role of hints in resolving unknown questions, the
dependence of hint quality on answer difficulty, and the feasibility of
employing automatic evaluation methods for hint assessment.