Video + Tracking

A video clip is a tensor of shape T × H × W × C (frames × height × width × channels). Neighbouring frames are nearly identical, which is both a burden (lots of redundant data) and a gift (strong continuity to exploit).

Tasks include action recognition (what is happening — often with 3-D convolutions or video transformers over short clips), optical flow (how every pixel moves) and multi-object tracking (MOT) — following each object with a stable identity across frames.

Most trackers use tracking-by-detection: run a detector on every frame, then link detections into tracks with stable ids.

SORT (Simple Online and Realtime Tracking) does the linking with two tools. First, a Kalman filter per track, with state (position, size, velocity):

• Predict — move the box forward assuming roughly constant velocity: x̂ = x + v·Δt.
• Update — when a matching detection arrives, blend prediction and measurement, weighted by their uncertainties.

Prediction lets a track keep moving for a few frames even when its detection is missing (brief occlusion or a missed detection).

Each frame, SORT builds a cost matrix between predicted track boxes (rows) and new detections (columns) — typically cost = 1 − IoU. The Hungarian algorithm finds the optimal one-to-one assignment that minimises total cost (in polynomial time), rather than greedily grabbing the best pair first.

After assignment:

• matched pairs update their tracks;
• pairs whose cost is too high (IoU below a gate) are rejected;
• unmatched detections start new tracks (new ids);
• unmatched tracks coast on their predictions and are deleted after a maximum age.

With motion and IoU alone, SORT fails when objects cross or are occluded for longer: predictions drift, nothing overlaps on reappearance, and the person gets a new id — an ID switch, the main error MOT metrics count (alongside misses and false positives; see MOTA, IDF1).

DeepSORT adds an appearance embedding from a re-identification CNN trained so that crops of the same person are close in feature space. The cost combines motion (Mahalanobis/IoU) and appearance distance, so a lost track can be re-identified after occlusion.

Optical flow estimates a motion vector for every pixel between two frames (no boxes). Deep models like RAFT compute it accurately; flow is used for motion analysis, video stabilisation and as an input feature for action recognition.