Benchmarking LLM powered Chatbots: Methods and Metrics
Debarag Banerjee Pooja Singh Arjun Avadhanam Saksham Srivastava
Autonomous conversational agents, i.e. chatbots, are becoming an increasingly
common mechanism for enterprises to provide support to customers and partners.
In order to rate chatbots, especially ones powered by Generative AI tools like
Large Language Models (LLMs) we need to be able to accurately assess their
performance. This is where chatbot benchmarking becomes important. In this
paper, we propose the use of a novel benchmark that we call the E2E (End to
End) benchmark, and show how the E2E benchmark can be used to evaluate accuracy
and usefulness of the answers provided by chatbots, especially ones powered by
LLMs. We evaluate an example chatbot at different levels of sophistication
based on both our E2E benchmark, as well as other available metrics commonly
used in the state of art, and observe that the proposed benchmark show better
results compared to others. In addition, while some metrics proved to be
unpredictable, the metric associated with the E2E benchmark, which uses cosine
similarity performed well in evaluating chatbots. The performance of our best
models shows that there are several benefits of using the cosine similarity
score as a metric in the E2E benchmark.