Knowledge-Aware Artifact Image Synthesis with LLM-Enhanced Prompting and
  Multi-Source Supervision
Shengguang Wu Zhenglun Chen Qi Su
Ancient artifacts are an important medium for cultural preservation and
restoration. However, many physical copies of artifacts are either damaged or
lost, leaving a blank space in archaeological and historical studies that calls
for artifact image generation techniques. Despite the significant advancements
in open-domain text-to-image synthesis, existing approaches fail to capture the
important domain knowledge presented in the textual description, resulting in
errors in recreated images such as incorrect shapes and patterns. In this
paper, we propose a novel knowledge-aware artifact image synthesis approach
that brings lost historical objects accurately into their visual forms. We use
a pretrained diffusion model as backbone and introduce three key techniques to
enhance the text-to-image generation framework: 1) we construct prompts with
explicit archaeological knowledge elicited from large language models (LLMs);
2) we incorporate additional textual guidance to correlated historical
expertise in a contrastive manner; 3) we introduce further visual-semantic
constraints on edge and perceptual features that enable our model to learn more
intricate visual details of the artifacts. Compared to existing approaches, our
proposed model produces higher-quality artifact images that align better with
the implicit details and historical knowledge contained within written
documents, thus achieving significant improvements across automatic metrics and
in human evaluation. Our code and data are available at
https://github.com/danielwusg/artifact_diffusion.