Segmentation

Segmentation assigns a label to every pixel. The output is a mask with the same height and width as the image.

• Semantic segmentation — a class per pixel; all sheep are just "sheep".
• Instance segmentation — separates individual objects: sheep 1, sheep 2. Mask R-CNN adds a small mask head to Faster R-CNN that predicts a mask inside each detected box.
• Panoptic segmentation — combines both: countable things (people, cars) get instance ids; amorphous stuff (sky, road, grass) gets semantic labels.

Applications range from medical imaging (tumour boundaries) to agriculture (field and crop mapping) to self-driving (drivable area).

A segmentation network has to answer two questions: what is in the image (needs large receptive fields and deep, semantic features) and where exactly (needs high resolution).

• The encoder repeatedly convolves and downsamples: features become more semantic, but spatial detail is lost — a 256 × 256 input may be 16 × 16 at the bottleneck.
• The decoder upsamples back to full resolution, using transposed convolutions ("learned upsampling") or interpolation + convolution.

U-Net's key idea is the skip connection: copy the encoder's high-resolution feature maps across and concatenate them with the decoder's upsampled maps at each level. The decoder then combines "what" from deep layers with "where" from shallow ones. Remove the skips and masks come out blurry, with poor boundaries.

With predicted mask P and ground truth G (sets of pixels):

• IoU (Jaccard) = |P ∩ G| / |P ∪ G| = TP / (TP + FP + FN).
• Dice (F1) = 2|P ∩ G| / (|P| + |G|) = 2TP / (2TP + FP + FN).

They are linked: Dice = 2·IoU / (1 + IoU), so Dice ≥ IoU (IoU 0.5 ↔ Dice 0.667). Both punish false positives (painting outside) and false negatives (missing parts).

Report per-class IoU and mean IoU (mIoU) across classes — pixel accuracy is misleading when background dominates: a model that predicts "background" everywhere can get 95% pixel accuracy on small lesions.

The standard loss is per-pixel cross-entropy. When the object of interest is tiny (a lesion covering 2% of the image), cross-entropy is dominated by easy background pixels.

Remedies:

• Dice loss (1 − Dice) optimises overlap directly and is insensitive to how much background there is — popular in medical imaging.
• Combined loss: cross-entropy + Dice, for stable gradients plus good overlap.
• Class weights or focal loss to emphasise rare classes.

Data augmentation must transform the image and mask identically (same crop, flip, rotation) — otherwise labels no longer line up with pixels.