Cache & Distil: Optimising API Calls to Large Language Models
Guillem Ramírez Matthias Lindemann Alexandra Birch Ivan Titov
Large-scale deployment of generative AI tools often depends on costly API
calls to a Large Language Model (LLM) to fulfil user queries. To curtail the
frequency of these calls, one can employ a smaller language model -- a student
-- which is continuously trained on the responses of the LLM. This student
gradually gains proficiency in independently handling an increasing number of
user requests, a process we term neural caching. The crucial element in neural
caching is a policy that decides which requests should be processed by the
student alone and which should be redirected to the LLM, subsequently aiding
the student's learning. In this study, we focus on classification tasks, and we
consider a range of classic active learning-based selection criteria as the
policy. Our experiments suggest that Margin Sampling and Query by Committee
bring consistent benefits across tasks and budgets.