Optimizing LLM Queries in Relational Workloads
Shu Liu Asim Biswal Audrey Cheng Xiangxi Mo Shiyi Cao Joseph E. Gonzalez Ion Stoica Matei Zaharia
Analytical database providers (e.g., Redshift, Databricks, BigQuery) have
rapidly added support for invoking Large Language Models (LLMs) through native
user-defined functions (UDFs) to help users perform natural language tasks,
such as classification, entity extraction, and translation, inside analytical
workloads. For instance, an analyst might want to extract customer sentiments
on millions of product reviews. However, LLM inference is highly expensive in
both computational and economic terms: for example, an NVIDIA L4 GPU running
Llama2-7B can only process 6 KB of text per second. In this paper, we explore
how to optimize LLM inference for analytical workloads that invoke LLMs within
relational queries. We show that relational queries present novel opportunities
for accelerating LLM inference, including reordering rows to maximize key-value
(KV) cache reuse within the LLM inference engine, reordering columns within a
row to further increase cache reuse, and deduplicating redundant inference
requests. We implement these optimizations in Apache Spark, with vLLM as the
model serving backend and achieve up to 4.4x improvement in end-to-end latency
on a benchmark of diverse LLM-based queries on real datasets. To the best of
our knowledge, this is the first work to explicitly address the problem of
optimizing LLM invocations within SQL queries.