Space Narrative: Generating Images and 3D Scenes of Chinese Garden from
  Text using Deep Learning
Jiaxi Shi1 Hao Hua1
The consistent mapping from poems to paintings is essential for the research
and restoration of traditional Chinese gardens. But the lack of firsthand
ma-terial is a great challenge to the reconstruction work. In this paper, we
pro-pose a method to generate garden paintings based on text descriptions using
deep learning method. Our image-text pair dataset consists of more than one
thousand Ming Dynasty Garden paintings and their inscriptions and post-scripts.
A latent text-to-image diffusion model learns the mapping from de-scriptive
texts to garden paintings of the Ming Dynasty, and then the text description of
Jichang Garden guides the model to generate new garden paintings. The cosine
similarity between the guide text and the generated image is the evaluation
criterion for the generated images. Our dataset is used to fine-tune the
pre-trained diffusion model using Low-Rank Adapta-tion of Large Language Models
(LoRA). We also transformed the generated images into a panorama and created a
free-roam scene in Unity 3D. Our post-trained model is capable of generating
garden images in the style of Ming Dynasty landscape paintings based on textual
descriptions. The gener-ated images are compatible with three-dimensional
presentation in Unity 3D.