Jigsaw: Supporting Designers in Prototyping Multimodal Applications by
  Assembling AI Foundation Models
David Chuan-En Lin Nikolas Martelaro
Recent advancements in AI foundation models have made it possible for them to
be utilized off-the-shelf for creative tasks, including ideating design
concepts or generating visual prototypes. However, integrating these models
into the creative process can be challenging as they often exist as standalone
applications tailored to specific tasks. To address this challenge, we
introduce Jigsaw, a prototype system that employs puzzle pieces as metaphors to
represent foundation models. Jigsaw allows designers to combine different
foundation model capabilities across various modalities by assembling
compatible puzzle pieces. To inform the design of Jigsaw, we interviewed ten
designers and distilled design goals. In a user study, we showed that Jigsaw
enhanced designers' understanding of available foundation model capabilities,
provided guidance on combining capabilities across different modalities and
tasks, and served as a canvas to support design exploration, prototyping, and
documentation.