How AI Uses Text to Generate Art

The idea behind generating art images based on text is to use a generative model that could generate images, while being conditioned by a representation of the text information. The generative models commonly used for it include Generative Adversarial Network (GAN), Variational Auto-Encoder (VAE), and Diffusion models. GANs include a Generator Neural Network model and a Discriminator model. The Generator learns to generate images that could be mistaken as real examples by the Discriminator model. The Discriminator simultaneously learns to classify real and generated images correctly. VAE models learn to encode the image into a latent space representation from which the original image could be reconstructed well using the Decoder model. The images are generated by sampling from the encoded latent space distribution. Diffusion models are inspired by thermodynamics and work by changing concentration through adding or removing noise. Denoising Diffusion models generate images by iteratively filling in the details of a noisy image representation.

OpenAI’s Contrastive Language-Image Pre-training (CLIP) model is being used widely for text-conditioned image synthesis. In CLIP, a Vision Transformer-based image encoder is jointly trained with a text encoder using image-text pairs and a multi-modal embedding space is learned, in which similar images and text are closer. These CLIP embeddings can be used for zero-shot reasoning in new tasks, by comparing images with text. CLIP is being used for conditioning and guiding different generative models.

In the original DALL-E model, a Transformer-based decoder was trained to generate images by taking a single stream of both text and image, while using a VAE. CLIP was used with it for re-ranking and choosing the best generated images. In DALL-E 2, a prior is generated using CLIP encoding and then a Diffusion model called unCLIP is used to reconstruct images from CLIP embedding by filling in details. In several models like VQGAN-CLIP and CLIP guided diffusion, the CLIP model is used to steer generative models like GANs and Diffusion models, towards generating output similar to given text. It’s done by finding similarity between CLIP embedding and generated images and using it as a loss function for optimization.

DreamStudio is based on the Stable Diffusion model which uses a Latent Diffusion Model (LDM). In LDM, a U-Net based Auto-Encoder is used to compress the latent space to facilitate generating high resolution images. The Diffusion model works on that latent space and also includes cross-attention conditioning mechanism. The diffusion is conditioned on text embeddings from CLIP.

Google Brain’s Imagen generates photorealistic images. It utilizes a large Transformer for text embeddings and then it uses a Diffusion model having U-Net and an improved sampling technique named dynamic thresholding.