CliqueParcel: An Approach For Batching LLM Prompts That Jointly
  Optimizes Efficiency And Faithfulness
Jiayi Liu Tinghan Yang Jennifer Neville
Large language models (LLMs) have become pivotal in recent research. However,
during the inference process, LLMs still require substantial resources. In this
paper, we propose CliqueParcel, a method designed to improve the efficiency of
LLMs via prompt batching. Existing strategies to optimize inference efficiency
often compromise on output quality, leading to a discounted output problem.
This issue might result in reduced accuracy or outputs that are less detailed.
CliqueParcel is our answer to this challenge. While ensuring accuracy and
minimizing deviations from the original outputs (i.e., faithfulness), our
method significantly improves efficiency during inference.
  To lay the groundwork, we first redefine efficiency measurements by excluding
the reduction in running time due to shorter lengths. Then, we provide a
comprehensive trade-off between efficiency and faithfulness to clarify the
nature of the 'discounted output' problem. Within the CliqueParcel framework,
we suggest multiple batching sub-methods and discuss the specific scenarios in
which they can be applied. During evaluation, CliqueParcel is tested on eight
widely recognized datasets, which can be classified into three types: reading
comprehension, open-source question-answering, and reasoning. Our experiments
explore the performance of CliqueParcel, including efficiency, faithfulness,
and the trade-off between them. This work provides novel insights into
inference efficiency and demonstrates promising performance.