CSC3104 · Revision guide

Computer Vision & Deep Learning on a few pages

The night-before sheet: every module's one-minute summary, traps, terms, formulas and the answer examiners want.

The 3-beat answer

Examiners reward structure. Every answer, about 30 seconds:

1 · ClaimOne sentence: what it is or does. State the rule directly.

2 · MechanismHow it works: the maths, the steps, the data structures.

3 · Trade-offWhen it fails, what it costs, or why the alternative breaks.

Formulas to know cold

Conv output size: O = ⌊(W − K + 2P) / S⌋ + 1
Conv parameters: (K · K · C_in + 1) · C_out
Normalisation: x′ = (x / 255 − μ) / σ
Cross-entropy: L = − Σ y_c · log p_c
Precision / Recall: P = TP/(TP+FP), R = TP/(TP+FN)
F1: F1 = 2PR / (P + R)
IoU: |A ∩ B| / |A ∪ B|
Dice: 2|A ∩ B| / (|A| + |B|) = 2·IoU / (1 + IoU)
Focal loss: FL = −α (1 − p_t)^γ · log p_t
ViT tokens: N = (H/P) · (W/P) (+1 for [CLS])
Self-attention: softmax(Q Kᵀ / √d_k) · V
Model size: params × bytes/param (FP32 = 4, FP16 = 2, INT8 = 1)
Quantisation: q = round(x / s) + z, s = (max − min) / 255

Module 1

Images as Data

An image is a tensor of numbers (H × W × C, values 0–255). Before training we scale to 0–1 and normalise per channel with the mean/std the backbone was trained on, so inputs live where the pretrained weights expect them. Kernels like Sobel are tiny weight grids that respond to intensity change, and augmentation creates new views of the same image so the model learns invariances instead of memorising.

Normalising with your own dataset's statistics when using a pretrained ImageNet backbone — use the backbone's mean/std.
Forgetting the ToTensor() step: Normalize expects 0–1 floats, not 0–255 integers.
Augmenting the validation/test set — augmentation is for training only.

Tensor (H, W, C): A multi-dimensional array. PyTorch uses (C, H, W); NumPy/OpenCV use (H, W, C) — and OpenCV loads as BGR!
Channel: One 2D slice of the image, e.g. the red values. Grayscale has 1, RGB has 3.
Normalisation: x′ = (x/255 − μ)/σ per channel. ImageNet μ ≈ (0.485, 0.456, 0.406), σ ≈ (0.229, 0.224, 0.225).
Kernel / filter: A small weight grid slid across the image; Sobel finds edges, Gaussian blurs.
Augmentation: Random label-preserving transforms applied only at training time.

Likely question: Why do we normalise image pixels using the ImageNet mean and standard deviation before feeding them into a pretrained CNN?

A pretrained network's weights were tuned on inputs normalised with ImageNet's per-channel mean and std. Normalising our images the same way puts them in the distribution the network expects, so early-layer activations sit in the range the filters were trained for. Mechanically, centring and scaling makes features comparable in magnitude, which keeps gradients well-behaved. If you skip it, the input distribution shifts and accuracy drops, or fine-tuning becomes unstable.

Module 2

Convolutional Neural Networks

CNNs replace giant fully connected layers with small shared kernels: local connectivity plus parameter sharing gives translation equivariance and far fewer weights. Pooling downsamples with zero learnable parameters and adds a little invariance. Know the output-size formula floor((W − K + 2P)/S) + 1 and the parameter count K·K·C_in·C_out + C_out.

Saying pooling has learnable weights — it has none.
Mixing up equivariance (output moves with input) and invariance (output doesn't change).
Forgetting the bias term (+C_out) or the input channels when counting conv parameters.

Receptive field: The region of the input that affects one output neuron; grows with depth.
Stride: How many pixels the kernel jumps each step. Stride 2 halves the size.
Padding: Zeros added around the border so edges get processed and size can be kept.
Feature map: The output of one filter across the image: ‘where is this pattern?’
Parameters per conv: (K·K·C_in + 1)·C_out — the +1 is the bias.

Likely question: Does max pooling have any learnable parameters? And what is the difference between translation equivariance and translation invariance?

Max pooling has zero learnable parameters: it's a fixed operation that outputs the maximum in each window, used to downsample and cut computation. Convolution is translation equivariant — shift the input and the feature map shifts by the same amount. Pooling adds local translation invariance — small shifts inside a window don't change the output. The trade-off is lost precise position information, which hurts dense tasks like segmentation.

Module 3

Training & Evaluation

Training is forward → loss → backprop → optimiser step, repeated over epochs. Tune on validation, touch test once, and guard against leakage. Read the curves: train falls but val rises means overfitting; both high means underfitting. On imbalanced data accuracy lies — use precision, recall, F1 and the confusion matrix.

Reporting accuracy on a 99%-negative dataset.
Forgetting model.eval(): BatchNorm and Dropout behave differently at inference.
Tuning hyper-parameters on the test set — that's leakage too.

Epoch: One full pass over the training set.
Overfitting: Train performance ≫ validation performance; the model memorised noise.
Precision: TP / (TP + FP) — how trustworthy a positive prediction is.
Recall: TP / (TP + FN) — how many real positives were found.
Data leakage: Test information sneaking into training, making scores unrealistically high.

Likely question: Why does Batch Normalization behave differently during training versus evaluation mode?

BatchNorm normalises activations using a mean and variance. In training it uses the statistics of the current mini-batch, and also updates running averages of them. In evaluation mode it uses those stored running averages instead, because at inference the batch may be a single image and predictions must be deterministic and independent of other samples. Forgetting model.eval() is a classic bug: outputs depend on the batch and accuracy drops.

Module 4

Transfer Learning

Reuse a backbone pretrained on a big dataset: early layers learn general edges and textures, only the head is task-specific. Small, similar data → freeze the backbone and train a linear probe. More or different data → fine-tune gently with small (discriminative) learning rates, unfreezing top-down, to avoid catastrophic forgetting.

Fine-tuning everything with a large learning rate and wiping out pretrained features.
Forgetting to set requires_grad=False (or to exclude frozen params from the optimiser).
Assuming ImageNet features transfer to very different domains (e.g. X-rays) without fine-tuning.

Backbone: The pretrained feature extractor (e.g. ResNet-50 without its final layer).
Head: The new task-specific layer(s) on top, e.g. Linear(2048 → num_classes).
Linear probe: Freeze everything; train only a linear classifier on the features.
Fine-tuning: Continue training some or all pretrained layers on the new data.
Domain shift: Source and target images look different (photos vs X-rays).

Likely question: What is catastrophic forgetting during fine-tuning, and under what dataset conditions would you freeze the backbone versus fine-tune the whole network?

Catastrophic forgetting is when fine-tuning with large updates overwrites the pretrained weights, destroying the general features that made transfer useful. If the target dataset is small and similar to the source, freeze the backbone and train only a new head — fewer trainable parameters means less overfitting. If the dataset is large or quite different (e.g. medical images), fine-tune more layers but with a small, often layer-wise learning rate. The trade-off is adaptation versus stability.

Module 5

Object Detection

Detection outputs class + confidence + box for every object. Two-stage detectors (Faster R-CNN) propose regions then refine them; one-stage detectors (YOLO) predict on a grid in one pass for real-time speed. IoU measures box overlap; NMS keeps the best box and removes overlapping duplicates above an IoU threshold; mAP evaluates the whole detector.

Computing IoU as intersection over the sum of areas (forgetting to subtract the intersection).
Running NMS across different classes when it should be per class.
Setting the NMS IoU threshold so low that neighbouring objects get suppressed.

Bounding box: (x, y, w, h) or (x1, y1, x2, y2) rectangle around an object.
IoU: Area of overlap / area of union between two boxes.
NMS: Greedy removal of duplicate boxes using confidence and IoU.
Anchor box: Pre-defined box shapes the detector refines into predictions.
mAP: Mean of per-class Average Precision (area under the precision–recall curve).

Likely question: Walk me through how Non-Maximum Suppression (NMS) works, and explain why it is necessary in object detection.

Detectors output many overlapping boxes for the same object, so NMS removes the duplicates. First drop boxes below a confidence threshold. Sort the rest by confidence, keep the highest, compute its IoU — intersection area over union area — with every remaining box of that class, and suppress any above a threshold like 0.5. Repeat with the next highest until none remain. Trade-off: too low a threshold deletes real objects in crowds; too high leaves duplicates.

Module 6

Segmentation

Segmentation labels every pixel. Semantic = class per pixel, instance = separate objects, panoptic = both. U-Net's encoder learns WHAT is there but loses WHERE; the decoder upsamples and skip connections bring back high-resolution detail for sharp boundaries. Evaluate with per-class IoU / mIoU or Dice = 2·overlap / (|A| + |B|).

Thinking Dice and IoU are the same number — Dice = 2·IoU / (1 + IoU) is always ≥ IoU.
Removing skip connections and expecting crisp edges.
Reporting pixel accuracy when the background dominates the image.

Semantic seg.: Class label per pixel; instances are not separated.
Instance seg.: Separate mask per object instance (e.g. Mask R-CNN).
Panoptic seg.: Things (instances) + stuff (semantic) in one output.
Dice: 2|A∩B| / (|A|+|B|) = 2·IoU / (1 + IoU).
Transposed conv: Learnable upsampling used in decoders (alt: bilinear + conv).

Likely question: Why are skip connections indispensable in U-Net? What happens to the output mask if you remove them?

The U-Net encoder downsamples with pooling to capture semantic context, but that discards fine spatial detail. Skip connections pass the high-resolution feature maps from each encoder level directly to the matching decoder level, where they're concatenated with the upsampled features. This gives the decoder both 'what' from deep layers and 'where' from shallow layers. Without them, the decoder must reconstruct detail from a coarse bottleneck, so masks come out blurry with inaccurate boundaries.

Module 7

ViTs + Multimodal

A ViT cuts the image into patches (224/16 → 14×14 = 196 tokens), projects each into an embedding, adds [CLS] and position embeddings, then uses self-attention so any patch can attend to any other. Weak inductive bias means ViTs need lots of data. CLIP learns a shared image–text space with a contrastive loss, enabling zero-shot classification by cosine similarity with text prompts.

Forgetting the +1 [CLS] token in the sequence length.
Saying ViTs are always better than CNNs — on small data CNN inductive bias usually wins.
Thinking CLIP zero-shot needs task-specific training — it only needs class prompts.

Patch embedding: Flatten a P×P×C patch and project it to a D-dim token.
[CLS] token: Extra learnable token whose final state is used for classification.
Self-attention: Each token mixes information from all tokens, weighted by Q·K similarity.
Inductive bias: Built-in assumptions (e.g. locality) that help learn from less data.
Contrastive loss: Pull matching image–text pairs together, push others apart.

Likely question: Why do Vision Transformers need significantly more training data than CNNs to reach peak performance? And how does CLIP perform zero-shot classification?

CNNs hard-code inductive biases — locality and translation equivariance — so they learn efficiently from less data. ViTs split the image into patch tokens and use global self-attention with very weak spatial bias, so they must learn spatial structure from data, which requires large-scale pretraining. CLIP learns a shared embedding space for images and text via contrastive training on image–caption pairs. For zero-shot classification, it embeds a text prompt per class, embeds the image, and predicts the class with the highest cosine similarity.

Module 8

Video + Tracking

Tracking-by-detection detects objects every frame and links them into tracks. SORT predicts each track's next position with a Kalman filter (constant velocity), builds a cost matrix such as 1 − IoU, and solves it optimally with the Hungarian algorithm. DeepSORT adds a re-ID appearance embedding so identities survive occlusions and crossings.

Greedy matching instead of the Hungarian algorithm's optimal one-to-one assignment.
Expecting IoU-only matching to keep IDs through occlusion.
Confusing optical flow (per-pixel motion) with box tracking.

Tracklet / track: A sequence of detections linked as one object over time.
Kalman filter: Predict → update loop for a state with uncertainty (here: position + velocity).
Hungarian algorithm: Optimal one-to-one assignment for a cost matrix, O(n³).
ID switch: Tracker swaps identities of two objects — a key MOT error.
Optical flow: Per-pixel motion field between consecutive frames.

Likely question: Explain the roles of the Kalman filter and the Hungarian algorithm in SORT. Why was DeepSORT developed?

In SORT, a Kalman filter keeps a motion state for each track — position, box size and velocity — predicts where it will be in the next frame, then updates with the matched detection. The Hungarian algorithm finds the optimal one-to-one assignment between predicted tracks and new detections using an IoU-based cost. SORT relies only on motion and overlap, so occlusions and crossings cause ID switches. DeepSORT adds a deep appearance (re-ID) embedding to the matching cost so identities survive occlusion.

Module 9

Deployment & Responsible Vision

Deployment means meeting latency, memory and power budgets on real hardware: quantise (FP32 → INT8 is 4× smaller), prune, and export to runtimes like ONNX, TensorRT or TFLite, then benchmark on the target device. Explain predictions with Grad-CAM and audit fairness with disaggregated per-group metrics, model cards and robustness tests.

Benchmarking on your laptop instead of the target device.
Reporting one overall accuracy number and hiding subgroup failures.
Confusing latency (time per request) with throughput (requests per second).

Quantisation: FP32 → INT8 (or FP16) weights/activations; ~4× smaller, faster on supporting hardware.
Pruning: Removing low-importance weights or channels.
ONNX: Open, framework-neutral model format for inference runtimes.
Latency vs throughput: Time per input vs inputs per second (batching trades one for the other).
Model card: Documentation of intended use, data, metrics by subgroup and limitations.

Likely question: How does Grad-CAM produce an interpretable visual explanation of a CNN prediction? And how do you audit a vision model for demographic bias?

Grad-CAM computes the gradient of the target class score with respect to the last convolutional layer's feature maps, global-average-pools those gradients into one importance weight per channel, takes the weighted sum of the feature maps and applies ReLU. Upsampled, this is a heatmap of the regions that pushed the prediction toward that class. To audit for bias, evaluate disaggregated metrics — accuracy, false positive and false negative rates — separately for each demographic subgroup, check the training data's representation, and document gaps; a high overall accuracy can hide large subgroup failures.