Diffusion idea exploration for art generation
Nikhil Verma
Cross-Modal learning tasks have picked up pace in recent times. With plethora
of applications in diverse areas, generation of novel content using multiple
modalities of data has remained a challenging problem. To address the same,
various generative modelling techniques have been proposed for specific tasks.
Novel and creative image generation is one important aspect for industrial
application which could help as an arm for novel content generation. Techniques
proposed previously used Generative Adversarial Network(GAN), autoregressive
models and Variational Autoencoders (VAE) for accomplishing similar tasks.
These approaches are limited in their capability to produce images guided by
either text instructions or rough sketch images decreasing the overall
performance of image generator. We used state of the art diffusion models to
generate creative art by primarily leveraging text with additional support of
rough sketches. Diffusion starts with a pattern of random dots and slowly
converts that pattern into a design image using the guiding information fed
into the model. Diffusion models have recently outperformed other generative
models in image generation tasks using cross modal data as guiding information.
The initial experiments for this task of novel image generation demonstrated
promising qualitative results.