Experimental Design for Active Transductive Inference in Large Language
  Models
Subhojyoti Mukherjee Ge Liu Aniket Deshmukh Anusha Lalitha Yifei Ma Branislav Kveton
Transduction, the ability to include query-specific examples in the prompt at
inference time, is one of the emergent abilities of large language models
(LLMs). In this work, we propose a framework for adaptive prompt design called
active transductive inference (ATI). We design the LLM prompt by adaptively
choosing few-shot examples for a given inference query. The examples are
initially unlabeled and we query the user to label the most informative ones,
which maximally reduces the uncertainty in the LLM prediction. We propose two
algorithms, GO and SAL, which differ in how the few-shot examples are chosen.
We analyze these algorithms in linear models: first GO and then use its
equivalence with SAL. We experiment with many different tasks and show that GO
and SAL outperform other methods for choosing few-shot examples in the LLM
prompt at inference time.