Transfer Learning

Training a modern vision model from scratch needs millions of labelled images and days of GPU time. Most projects have hundreds or a few thousand. Transfer learning starts from a network already trained on a large dataset (ImageNet: 1.2 M images, 1,000 classes) and adapts it.

It works because features are hierarchical: early layers learn edges, colours and textures that are useful for almost any image; middle layers learn patterns and parts; only the last layers are specific to the original 1,000 classes. So we keep the general part (the backbone) and replace the specific part (the head).

Feature extraction / linear probe — freeze the backbone and train only a new head:

model = resnet50(weights="IMAGENET1K_V2")
for p in model.parameters():
    p.requires_grad = False
model.fc = nn.Linear(2048, num_classes)        # the only trainable layer

Fast, needs little data, can't overfit much — but limited if your images differ a lot from ImageNet.

Fine-tuning — unfreeze some or all of the backbone and keep training with a small learning rate, so the features adapt to your domain.

If you fine-tune with a large learning rate, the first batches produce large, noisy gradients (the new head is random), which overwrite the useful pretrained features — catastrophic forgetting. Validation accuracy can end up worse than the linear probe.

Good practice:

1. Train the head first with the backbone frozen until it is reasonable.
2. Unfreeze from the top down — last block first, earlier blocks later if needed.
3. Use discriminative learning rates: small for early layers (1e-5), larger for later layers and the head (1e-4 to 1e-3).
4. Use warm-up, weight decay and strong augmentation; keep BatchNorm layers in eval mode when batches are tiny.

Domain shift — when your images look different from the pretraining data (grayscale X-rays, multispectral satellite tiles, microscope slides) — reduces how much transfers. The further the domain, the more layers you need to fine-tune, and the more a domain-specific pretrained model helps (e.g. one pretrained on medical images).

Today's backbones are foundation models trained with self-supervision on huge unlabelled collections — e.g. DINOv2, or CLIP's image encoder. Their features are general enough that a simple linear head often matches full fine-tuning for classification, and the same features serve segmentation and depth estimation. Always try a linear probe on a foundation model as a strong, cheap baseline.