How Multimodal Integration Boost the Performance of LLM for
  Optimization: Case Study on Capacitated Vehicle Routing Problems
Yuxiao Huang Wenjie Zhang Liang Feng Xingyu Wu Kay Chen Tan
Recently, large language models (LLMs) have notably positioned them as
capable tools for addressing complex optimization challenges. Despite this
recognition, a predominant limitation of existing LLM-based optimization
methods is their struggle to capture the relationships among decision variables
when relying exclusively on numerical text prompts, especially in
high-dimensional problems. Keeping this in mind, we first propose to enhance
the optimization performance using multimodal LLM capable of processing both
textual and visual prompts for deeper insights of the processed optimization
problem. This integration allows for a more comprehensive understanding of
optimization problems, akin to human cognitive processes. We have developed a
multimodal LLM-based optimization framework that simulates human
problem-solving workflows, thereby offering a more nuanced and effective
analysis. The efficacy of this method is evaluated through extensive empirical
studies focused on a well-known combinatorial optimization problem, i.e.,
capacitated vehicle routing problem. The results are compared against those
obtained from the LLM-based optimization algorithms that rely solely on textual
prompts, demonstrating the significant advantages of our multimodal approach.