OpenAI unveils GLIDE, a neural network that edits images from text descriptions
Text-to-image generation was one of the most active and exciting areas of artificial intelligence in 2021. In January 2021, OpenAI unveiled DALL-E, a version of its GPT-3 transformer language model with 12 billion parameters, designed to create photorealistic images from text captions used as prompts.
DALL-E's instant success in the AI community and its impressive performance also drew wide media coverage. In November 2021, tech giant NVIDIA released GauGAN2, a GAN-based model. The name was inspired by French post-impressionist painter Paul Gauguin, while DALL-E was inspired by surrealist artist Salvador Dalí.
Not to be outdone, OpenAI researchers introduced GLIDE (Guided Language-to-Image Diffusion for Generation and Editing), a neural network that makes DALL-E competitive in performance while using less than one-third of the parameters.
"Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique that trades off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples," says the research published on arXiv.
While most images are relatively easy to describe in words, creating images from text input requires special skills and many hours of work. Using AI to automatically create photorealistic images from natural-language prompts not only lets people produce rich and diverse visual content with unprecedented ease, but also enables easier iterative refinement and detailed control over generated images.
Recent research has shown that likelihood-based diffusion models can also produce high-quality synthetic images, especially when combined with a guidance technique designed to trade off diversity for fidelity. In May 2021, OpenAI introduced a guided model that lets algorithms depend on classifier labels.
GLIDE builds on this progress by applying guided diffusion to text-conditional image synthesis. After training a GLIDE model with 3.5 billion parameters that uses a text encoder to condition on natural-language descriptions, the researchers compared two different guidance strategies: CLIP guidance and classifier-free guidance.
CLIP is a scalable approach to learning joint representations of text and images that provides a score reflecting how closely an image matches a caption. The team applied this method to its models by replacing the classifier with a CLIP model that "guides" the network.
Classifier-free guidance is a method for steering diffusion models that does not require training a separate classifier. It has two appealing properties:
- It allows a single model to use its own knowledge during guidance, rather than relying on a separate classification model.
- It simplifies guidance when the information is difficult to predict with a classifier.
The researchers noted that human evaluators preferred images produced with classifier-free guidance for both photorealism and caption similarity.
In tests, GLIDE produced high-quality images with realistic shadows, reflections and textures. The model can also combine several concepts (for example, corgis, bow ties and party hats), attaching attributes such as colors to these objects.
Besides generating images from text, GLIDE can also be used to edit existing images: inserting new objects, adding shadows and reflections, and performing inpainting, all through natural-language text prompts.
GLIDE can also turn simple line sketches into photorealistic images, and its capabilities for zero-shot generation and inpainting in complex scenarios are very strong.
Compared with DALL-E, human evaluators preferred GLIDE's output images, even though it is a much smaller model (3.5 billion parameters versus 12 billion). It requires lower sampling latency and does not need CLIP reranking.
The research team recognizes that their neural network could make it easier for malicious actors to create convincing disinformation or deepfakes. To protect against such uses, they released a smaller noised CLIP model and a diffusion network trained on filtered datasets. The source code is available on the project's GitHub.
"GLIDE enables creating photorealistic images, combining concepts in the right way, and performing artistic rendering. According to the team's estimates, the results look preferable to DALL-E samples in 87% of cases," OpenAI said.
Reviewers at ZME Science noted that the quality of the final images is often comparable to painstaking work in Photoshop. The system handles complex prompts such as "hang a Van Gogh painting on the wall" but does not always understand strange requests like "a car with triangular wheels."
