Towards Optimizing the Costs of LLM Usage
Shivanshu Shekhar Tanishq Dubey Koyel Mukherjee Apoorv Saxena Atharv Tyagi Nishanth Kotla
Generative AI and LLMs in particular are heavily used nowadays for various
document processing tasks such as question answering and summarization.
However, different LLMs come with different capabilities for different tasks as
well as with different costs, tokenization, and latency. In fact, enterprises
are already incurring huge costs of operating or using LLMs for their
respective use cases.
  In this work, we propose optimizing the usage costs of LLMs by estimating
their output quality (without actually invoking the LLMs), and then solving an
optimization routine for the LLM selection to either keep costs under a budget,
or minimize the costs, in a quality and latency aware manner. We propose a
model to predict the output quality of LLMs on document processing tasks like
summarization, followed by an LP rounding algorithm to optimize the selection
of LLMs. We study optimization problems trading off the quality and costs, both
theoretically and empirically. We further propose a sentence simplification
model for reducing the number of tokens in a controlled manner. Additionally,
we propose several deterministic heuristics for reducing tokens in a quality
aware manner, and study the related optimization problem of applying the
heuristics optimizing the quality and cost trade-off. We perform extensive
empirical validation of our methods on not only enterprise datasets but also on
open-source datasets, annotated by us, and show that we perform much better
compared to closest baselines. Our methods reduce costs by 40%- 90% while
improving quality by 4%-7%. We will release the annotated open source datasets
to the community for further research and exploration.