LLM as an Art Director (LaDi): Using LLMs to improve Text-to-Media
  Generators
Allen Roush Emil Zakirov Artemiy Shirokov Polina Lunina Jack Gane Alexander Duffy Charlie Basil Aber Whitcomb Jim Benedetto Chris DeWolfe
Recent advancements in text-to-image generation have revolutionized numerous
fields, including art and cinema, by automating the generation of high-quality,
context-aware images and video. However, the utility of these technologies is
often limited by the inadequacy of text prompts in guiding the generator to
produce artistically coherent and subject-relevant images. In this paper, We
describe the techniques that can be used to make Large Language Models (LLMs)
act as Art Directors that enhance image and video generation. We describe our
unified system for this called "LaDi". We explore how LaDi integrates multiple
techniques for augmenting the capabilities of text-to-image generators (T2Is)
and text-to-video generators (T2Vs), with a focus on constrained decoding,
intelligent prompting, fine-tuning, and retrieval. LaDi and these techniques
are being used today in apps and platforms developed by Plai Labs.