ViT tokens & CLIP zero-shot
Count the tokens a Vision Transformer actually processes and what attention costs, then turn CLIP's similarities into zero-shot class probabilities.
Predict first
Commit to a guess before you touch anything. Being wrong here is how it sticks.
A ViT on 224×224 images with 16×16 patches and a [CLS] token. Sequence length?
Experiment
Move the controls and watch the outputs. Goals tick off automatically when you reach them.
Patches & tokens
CLIP zero-shot
Explain it
Put what you saw into words. This is what the exam will ask you to do.
Why do ViTs need more data than CNNs, and how does CLIP classify images it was never trained to classify?
Code it
Write real code against real data. Python and SQL run right here in your browser and hidden tests check your work; the AI reviews it like an examiner.
Scaled dot-product attention
Implement softmax(x, axis=-1) (numerically stable: subtract the max) and
attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V returning (output, weights) for N × d arrays.
Also write vit_tokens(image_size, patch) → sequence length including [CLS].
Challenge
Fixed variations, a new problem every day, and unlimited AI challenges built from your current settings.
ViT-Base Sequence Length
A ViT takes 224×224 images with 16×16 patches and prepends 1 [CLS] token. What is the total token sequence length L?