Thrust: Adaptively Propels Large Language Models with External Knowledge
Xinran Zhao Hongming Zhang Xiaoman Pan Wenlin Yao Dong Yu Jianshu Chen
Although large-scale pre-trained language models (PTLMs) are shown to encode
rich knowledge in their model parameters, the inherent knowledge in PTLMs can
be opaque or static, making external knowledge necessary. However, the existing
information retrieval techniques could be costly and may even introduce noisy
and sometimes misleading knowledge. To address these challenges, we propose the
instance-level adaptive propulsion of external knowledge (IAPEK), where we only
conduct the retrieval when necessary. To achieve this goal, we propose
measuring whether a PTLM contains enough knowledge to solve an instance with a
novel metric, Thrust, which leverages the representation distribution of a
small number of seen instances. Extensive experiments demonstrate that thrust
is a good measurement of PTLM models' instance-level knowledgeability.
Moreover, we can achieve significantly higher cost-efficiency with the Thrust
score as the retrieval indicator than the naive usage of external knowledge on
88% of the evaluated tasks with 26% average performance improvement. Such
findings shed light on the real-world practice of knowledge-enhanced LMs with a
limited knowledge-seeking budget due to computation latency or costs.