Zero-Shot Multi-task Hallucination Detection
Patanjali Bhamidipati Advaith Malladi Manish Shrivastava Radhika Mamidi
In recent studies, the extensive utilization of large language models has
underscored the importance of robust evaluation methodologies for assessing
text generation quality and relevance to specific tasks. This has revealed a
prevalent issue known as hallucination, an emergent condition in the model
where generated text lacks faithfulness to the source and deviates from the
evaluation criteria. In this study, we formally define hallucination and
propose a framework for its quantitative detection in a zero-shot setting,
leveraging our definition and the assumption that model outputs entail task and
sample specific inputs. In detecting hallucinations, our solution achieves an
accuracy of 0.78 in a model-aware setting and 0.61 in a model-agnostic setting.
Notably, our solution maintains computational efficiency, requiring far less
computational resources than other SOTA approaches, aligning with the trend
towards lightweight and compressed models.