DeLLMa: A Framework for Decision Making Under Uncertainty with Large
  Language Models
Ollie Liu Deqing Fu Dani Yogatama Willie Neiswanger
Large language models (LLMs) are increasingly used across society, including
in domains like business, engineering, and medicine. These fields often grapple
with decision-making under uncertainty, a critical yet challenging task. In
this paper, we show that directly prompting LLMs on these types of
decision-making problems yields poor results, especially as the problem
complexity increases. To overcome this limitation, we propose DeLLMa
(Decision-making Large Language Model assistant), a framework designed to
enhance decision-making accuracy in uncertain environments. DeLLMa involves a
multi-step scaffolding procedure, drawing upon principles from decision theory
and utility theory, to provide an optimal and human-auditable decision-making
process. We validate our framework on decision-making environments involving
real agriculture and finance data. Our results show that DeLLMa can
significantly improve LLM decision-making performance, achieving up to a 40%
increase in accuracy over competing methods.