Model Generation from Requirements with LLMs: an Exploratory Study
Alessio Ferrari Sallam Abualhaija Chetan Arora
Complementing natural language (NL) requirements with graphical models can
improve stakeholders' communication and provide directions for system design.
However, creating models from requirements involves manual effort. The advent
of generative large language models (LLMs), ChatGPT being a notable example,
offers promising avenues for automated assistance in model generation. This
paper investigates the capability of ChatGPT to generate a specific type of
model, i.e., UML sequence diagrams, from NL requirements. We conduct a
qualitative study in which we examine the sequence diagrams generated by
ChatGPT for 28 requirements documents of various types and from different
domains. Observations from the analysis of the generated diagrams have
systematically been captured through evaluation logs, and categorized through
thematic analysis. Our results indicate that, although the models generally
conform to the standard and exhibit a reasonable level of understandability,
their completeness and correctness with respect to the specified requirements
often present challenges. This issue is particularly pronounced in the presence
of requirements smells, such as ambiguity and inconsistency. The insights
derived from this study can influence the practical utilization of LLMs in the
RE process, and open the door to novel RE-specific prompting strategies
targeting effective model generation.