Hallucination-minimized Data-to-answer Framework for Financial
  Decision-makers
Sohini Roychowdhury Andres Alvarez Brian Moore Marko Krema Maria Paz Gelpi Federico Martin Rodriguez Angel Rodriguez Jose Ramon Cabrejas Pablo Martinez Serrano Punit Agrawal Arijit Mukherjee
Large Language Models (LLMs) have been applied to build several automation
and personalized question-answering prototypes so far. However, scaling such
prototypes to robust products with minimized hallucinations or fake responses
still remains an open challenge, especially in niche data-table heavy domains
such as financial decision making. In this work, we present a novel
Langchain-based framework that transforms data tables into hierarchical textual
data chunks to enable a wide variety of actionable question answering. First,
the user-queries are classified by intention followed by automated retrieval of
the most relevant data chunks to generate customized LLM prompts per query.
Next, the custom prompts and their responses undergo multi-metric scoring to
assess for hallucinations and response confidence. The proposed system is
optimized with user-query intention classification, advanced prompting, data
scaling capabilities and it achieves over 90% confidence scores for a variety
of user-queries responses ranging from {What, Where, Why, How, predict, trend,
anomalies, exceptions} that are crucial for financial decision making
applications. The proposed data to answers framework can be extended to other
analytical domains such as sales and payroll to ensure optimal hallucination
control guardrails.