ViTs + Multimodal

A Vision Transformer (ViT) reuses the transformer architecture from language models by turning an image into a sequence of tokens:

1. Cut the image into P × P patches: a 224 × 224 image with 16 × 16 patches gives a 14 × 14 grid = 196 patches.
2. Flatten each patch (16 × 16 × 3 = 768 values) and multiply by a learned matrix — the patch embedding — to get a D-dimensional token.
3. Prepend a learnable [CLS] token whose final state is used for classification → sequence length 197.
4. Add position embeddings so the model knows where each patch came from (otherwise the order of patches would be meaningless to it).

The sequence then passes through standard transformer blocks: multi-head self-attention, an MLP, layer norm and residual connections.

Each token produces a query (what am I looking for?), a key (what do I contain?) and a value (what do I pass on?) by linear projections.

Attention(Q, K, V) = softmax(QKᵀ / √d) · V

• QKᵀ scores every pair of tokens; dividing by √d keeps the scores in a range where softmax still has useful gradients.
• softmax turns each row into weights that sum to 1.
• Each token's output is a weighted mix of all values.

So a patch of a dog's ear can attend directly to the tail on the other side of the image in one layer — a CNN needs many layers for its receptive field to grow that large. The cost: the attention matrix is N × N, so compute and memory grow quadratically with the number of tokens. Halving the patch size gives 4× the tokens and ~16× the attention cost.

CNNs have strong built-in assumptions — locality and translation equivariance — so they learn efficiently from modest data. ViTs have weak inductive bias: nothing tells them that neighbouring patches are related; they must learn it from data (position embeddings start random).

Consequences:

• Trained from scratch on ImageNet-1k alone, early ViTs underperformed CNNs.
• Pretrained on very large datasets (ImageNet-21k, JFT-300M) or with self-supervision, the flexibility wins: ViTs match or beat CNNs and scale better.
• In practice you almost always fine-tune a pretrained ViT rather than train one; with little data and no pretraining, a CNN is the safer choice.

Hybrid designs (convolutional stems, Swin Transformer's local windows) put some locality back to get the best of both.

CLIP trains two encoders together — an image encoder (ViT or ResNet) and a text encoder — on ~400 million image–caption pairs from the web. The contrastive loss: in a batch of N pairs, maximise cosine similarity for the N matching pairs and minimise it for the N² − N mismatched ones. The result is a shared embedding space for images and text.

Zero-shot classification — no task-specific training:

1. Write a prompt per class: "a photo of a dog", "a photo of a cat", …
2. Embed the prompts and the image.
3. Compute cosine similarities, multiply by a learned scale (≈100) and apply softmax → class probabilities.

Prompt wording matters ("a satellite photo of farmland"), and CLIP inherits biases and blind spots from web data — always evaluate on your own domain before relying on it.