Large Language and Text-to-3D Models for Engineering Design Optimization
Thiago Rios Stefan Menzel Bernhard Sendhoff
The current advances in generative AI for learning large neural network
models with the capability to produce essays, images, music and even 3D assets
from text prompts create opportunities for a manifold of disciplines. In the
present paper, we study the potential of deep text-to-3D models in the
engineering domain, with focus on the chances and challenges when integrating
and interacting with 3D assets in computational simulation-based design
optimization. In contrast to traditional design optimization of 3D geometries
that often searches for the optimum designs using numerical representations,
such as B-Spline surface or deformation parameters in vehicle aerodynamic
optimization, natural language challenges the optimization framework by
requiring a different interpretation of variation operators while at the same
time may ease and motivate the human user interaction. Here, we propose and
realize a fully automated evolutionary design optimization framework using
Shap-E, a recently published text-to-3D asset network by OpenAI, in the context
of aerodynamic vehicle optimization. For representing text prompts in the
evolutionary optimization, we evaluate (a) a bag-of-words approach based on
prompt templates and Wordnet samples, and (b) a tokenisation approach based on
prompt templates and the byte pair encoding method from GPT4. Our main findings
from the optimizations indicate that, first, it is important to ensure that the
designs generated from prompts are within the object class of application, i.e.
diverse and novel designs need to be realistic, and, second, that more research
is required to develop methods where the strength of text prompt variations and
the resulting variations of the 3D designs share causal relations to some
degree to improve the optimization.