DesignQA: A Multimodal Benchmark for Evaluating Large Language Models'
  Understanding of Engineering Documentation
Anna C. Doris Daniele Grandi Ryan Tomich Md Ferdous Alam Hyunmin Cheong Faez Ahmed
This research introduces DesignQA, a novel benchmark aimed at evaluating the
proficiency of multimodal large language models (MLLMs) in comprehending and
applying engineering requirements in technical documentation. Developed with a
focus on real-world engineering challenges, DesignQA uniquely combines
multimodal data-including textual design requirements, CAD images, and
engineering drawings-derived from the Formula SAE student competition.
Different from many existing MLLM benchmarks, DesignQA contains
document-grounded visual questions where the input image and input document
come from different sources. The benchmark features automatic evaluation
metrics and is divided into segments-Rule Comprehension, Rule Compliance, and
Rule Extraction-based on tasks that engineers perform when designing according
to requirements. We evaluate state-of-the-art models like GPT4 and LLaVA
against the benchmark, and our study uncovers the existing gaps in MLLMs'
abilities to interpret complex engineering documentation. Key findings suggest
that while MLLMs demonstrate potential in navigating technical documents,
substantial limitations exist, particularly in accurately extracting and
applying detailed requirements to engineering designs. This benchmark sets a
foundation for future advancements in AI-supported engineering design
processes. DesignQA is publicly available at:
https://github.com/anniedoris/design_qa/.