CLIP: The Paper That Made Vision and Language Speak the Same Language

OpenAI's 2021 contrastive image-text pretraining paper. How CLIP's joint embedding space enabled zero-shot image classification, DALL-E, Stable Diffusion, and GPT-4V.

Before CLIP, teaching a model to understand images required labelled datasets — millions of images annotated by human labellers. The bottleneck was always the same: annotation was expensive and hard to scale beyond a few thousand categories.

In January 2021, OpenAI published 'Learning Transferable Visual Models from Natural Language Supervision'. CLIP was trained on 400 million image-text pairs from the internet with no human annotation labels, learning to align images and text in a shared embedding space. CLIP is the foundation of multimodal AI — DALL-E 2, Stable Diffusion, and GPT-4V all build on it.

Contrastive pretraining: learning from text

CLIP's training objective: given a batch of N image-text pairs, maximise similarity between the N correct pairings and minimise similarity between all N²−N incorrect pairings. The result: an image encoder and a text encoder that share a vector space where matching pairs are close and non-matching pairs are far apart.

Zero-shot image classification

Without task-specific training, CLIP matched models trained directly on ImageNet. Mechanism: encode the image, encode text templates ('a photo of a dog', 'a photo of a cat') — classify by highest cosine similarity. On ImageNet, CLIP achieved 76.2% top-1 accuracy zero-shot, matching ResNet-101 trained on 1.28M labelled examples.

Why CLIP became the foundation of generative AI

• DALL-E 2: uses CLIP image embeddings to guide diffusion generation — text prompt → CLIP embedding → diffusion model
• Stable Diffusion: uses CLIP's frozen text encoder to encode prompts, injected into the UNet
• GPT-4V and LLaVA: use CLIP or CLIP-like visual encoders to produce image tokens for the language model
• Multimodal search: CLIP embeddings enable semantic image search from text queries — and vice versa

CLIP's limitations

• OCR and spatial reasoning: struggles with reading text in images and understanding precise spatial relationships
• Counting: cannot reliably count objects in images
• Caption quality: web captions are noisy and often describe images at a high level without useful detail
• Bias: inherits biases from internet image-caption pairs — demographic, geographic, cultural

Explore multimodal AI and vision-language models →: See how CLIP-style embeddings power search and understanding tasks.