Leeroo Orchestrator: Elevating LLMs Performance Through Model
  Integration
Alireza Mohammadshahi Ali Shaikh Majid Yazdani
In this paper, we propose an architecture to harness the collective knowledge
of multiple trained LLMs to create a new state-of-the-art. At the core of this
framework is a LLM-based orchestrator that is adept at picking the right
underlying LLM experts for optimal task execution. Inspired by self-play in
reinforcement learning, we created a loop of query generation, orchestration,
and evaluation to generate training data for the orchestrator. Our evaluation
focused on the MMLU benchmark, employing models with 7B, 13B, and 34B
parameters available on Hugging Face. The results demonstrate new
state-of-the-art open-source models: Our Leeroo orchestrator achieves
performance on par with the Mixtral model while incurring only two-thirds of
its cost. Moreover, increasing the allowed cost surpasses Mixtral's accuracy by
over 5% at the same cost level, reaching an accuracy of 75.9%. Further
enhancements were observed when integrating GPT4 into the underlying model
pool. The Leeroo orchestrator nearly matches GPT4's performance at half the
cost and even exceeds GPT4's results with a 25% cost reduction. These findings
illustrate the potential of our architecture in creating state-of-the-art and
cost-effective LLMs by optimizing the synergy between multiple LLMs to achieve
superior performance outcomes.