CLIP: How Contrastive Vision-Language Pretraining Works

OpenAI's CLIP trained on 400M image-text pairs using contrastive loss — and changed what AI can do with vision. How the dual encoder works, what zero-shot classification means, and why CLIP embeddings power most production image search today.

In 2021, OpenAI published CLIP — Contrastive Language-Image Pre-Training — and quietly changed what AI could do with images. Before CLIP, image classifiers were trained on fixed label sets. ImageNet could tell you if a photo contained a dog or a cat, but only those categories it had seen in training. CLIP broke that constraint: it could classify images into categories it had never seen during training, just from a text description.

The trick was in both the training data and the training objective. Previous vision models trained on curated datasets with human-written labels. CLIP trained on 400 million (image, text) pairs scraped from the internet — product pages, news articles, social media posts — where the text was naturally associated with the image. The signal was noisy, but the scale made up for it.

The Dual Encoder Architecture

CLIP trains two encoders jointly: a vision encoder (a ViT or ResNet) that maps images to embeddings, and a text encoder (a transformer) that maps text to embeddings. Both output vectors in the same shared embedding space — 512 or 768 dimensions depending on model size.

The training objective is contrastive learning. For a batch of N (image, text) pairs, CLIP computes the N×N matrix of cosine similarities between all image and text embeddings. The goal: the N diagonal entries (the matched pairs) should have high similarity, and the N²-N off-diagonal entries (the mismatched pairs) should have low similarity.

Zero-Shot Classification

At inference time, zero-shot classification works by encoding candidate labels as text: 'a photo of a dog', 'a photo of a cat', 'a photo of a car'. You embed both the image and all the label strings, then pick the label whose text embedding is most similar to the image embedding. No fine-tuning required — the labels are just text.

On ImageNet, CLIP's zero-shot classification matched the accuracy of a supervised ResNet-50 trained explicitly on ImageNet. It did this without ever seeing ImageNet during training. That's the headline result — and it's why CLIP was a paradigm shift.

Why CLIP Embeddings Power Production Image Search

CLIP embeddings are cross-modal — a text query and a matching image produce nearby vectors. This makes them directly useful for image search: embed a text query, find nearest-neighbor images in a CLIP embedding index. No need for alt text or human-written image descriptions. The image semantics are already in the embedding.

Most production reverse image search, product discovery pipelines, and content moderation systems today use CLIP embeddings (or CLIP derivatives like SigLIP, OpenCLIP, or ALIGN) as the backbone. The embedding space is rich enough to capture semantic similarity, not just visual similarity — 'a red sports car' and 'Ferrari on a racetrack' end up close together.

Limitations

• Weak on fine-grained counting and spatial reasoning ('two dogs to the left of a cat' is hard).
• Struggles with text rendered in images — OCR-like tasks are not CLIP's strength.
• The text encoder has a 77-token context limit — long captions are truncated.
• Zero-shot performance degrades for highly specialized domains (medical imaging, satellite imagery) without domain-specific fine-tuning.
• Sensitive to prompt phrasing — 'a photo of a [class]' consistently outperforms bare class names.

CLIP Derivatives Worth Knowing

Explore Embedding Space →: See how embeddings cluster semantically in the Embedding Space module.