CSC3119 · Module 6 of 9

Serving Architectures, Load Balancing & Canary Traffic Routing

Implement zero-downtime upgrades, blue-green environments, and canary error budgets

~56 min to study properly
0%
Open lab

In mission-critical hospital systems, you cannot take an AI prediction service offline for 15 minutes whenever an updated model checkpoint is released. We require zero-downtime deployment strategies.

Checkpoint · the examiner asks

Compare Blue/Green and Canary deployment strategies for machine learning models. How do you configure health checks and error budgets to automate rollbacks?

In one minute

Zero-downtime releases put a load balancer in front of healthy replicas. Blue/Green flips all traffic from v1 to a verified v2 at once; Canary sends a small slice (e.g. 5%) to v2 and watches error rate, p99 latency and prediction anomalies, rolling back automatically when the error budget is breached. Canary limits the blast radius of a bad model.

Traps examiners look for
  • Thinking Blue/Green limits blast radius — it switches 100% at once.
  • Rolling out a canary without automated rollback thresholds.
  • Watching only HTTP errors and missing silent prediction anomalies.
Read

Study notes

01Zero-downtime as a requirement

A hospital triage service can't go offline for 15 minutes every time a new model is released. Releases must be zero-downtime, and a bad model must be undone in seconds, not after a bug report.

The building block is a load balancer / reverse proxy (Nginx, Traefik, an AWS ALB) in front of several replicas of the service. It sends each request to a healthy replica and can split traffic between versions. Replicas report readiness with a readiness probe (/health), so traffic only goes to instances that have loaded the model.

02Blue/Green deployment

Run two identical environments. Blue serves all traffic with v1. Deploy v2 to Green, run health checks and smoke tests against it, then switch the router so 100% of traffic goes to Green. Keep Blue running so you can switch back instantly.

  • ✓ Simple and instant; rollback is just switching back.
  • ✗ Every user hits v2 at the same moment, so the blast radius of a bug is 100%. Problems that only appear under real traffic — rare inputs, load — affect everyone before you notice.
  • ✗ Needs double the capacity during the switch.

03Canary releases

Named after canaries in coal mines: expose a small slice of real traffic to v2 first — say 5% — while 95% stays on v1. Watch it closely. If it's healthy, increase step by step (5 → 25 → 50 → 100%). If not, send its traffic back to 0%.

What to watch, compared with v1 over the same period:

  • Error rate (HTTP 5xx) and timeouts.
  • Latency, especially p99.
  • Prediction behaviour — distribution of predicted risk, share of high-risk flags. A model can be "healthy" by HTTP standards and still clinically wrong.

The thresholds form an error budget: e.g. roll back automatically if the canary's 5xx rate exceeds 1.5% or p99 exceeds 300 ms. Automating this is the point — a human might take minutes; a router takes seconds.

04Shadow, A/B and choosing a strategy

Two more patterns worth knowing:

  • Shadow (mirrored) deployment — v2 receives a copy of live traffic but its answers are discarded. Zero user risk; good for comparing predictions with v1 before any exposure.
  • A/B testing — split traffic to measure which version produces better outcomes (not just fewer errors), usually for longer, with statistical tests.

For clinical models a sensible path is shadow → canary with automated rollback → full rollout, with the old version kept warm until the new one has a track record.

Watch it done

Worked examples

Worked example 1

Errors during a canary

A gateway sends 5% of 240 QPS to v2. Over 15 minutes v2 has a 2.5% error rate. How many failed requests, and what would a 1.5% error budget do?

    Worked example 2

    Is the canary actually worse?

    Over one hour v1 had 18 errors in 100,000 requests (0.018%) and the canary 4 errors in 5,000 (0.08%). Both are under a 1.5% budget. Should the rollout continue?

      Vocabulary

      Key terms

      Shadow Deployments (Dark Traffic)

      In ultra-high stakes domains like medical diagnosis or autonomous driving, companies use Shadow Deployments: real production requests are asynchronously mirrored to the new model in the background, testing predictions against ground truth without exposing end users to any risk.

      Practise

      Quick check

      0 of 8 correct so far. Wrong answers come with an explanation — and the buddy can unpack them.

      Q1An ingress router directs 10% of traffic to a new model v2. Within 5 minutes, v2 has an error rate of 8% while v1 has 0.05%. What should happen?

      Q2What is the primary difference between a Liveness Probe and a Readiness Probe in container orchestration?

      Q3What is the blast radius of a Blue/Green switch?

      Q4In a canary release, what share of traffic usually goes to v2 first?

      Q5Which signal should NOT be the only canary health check?

      Q6What does a shadow deployment do with v2's responses?

      Q710% of 200 QPS to a canary with a 3% error rate for 10 minutes. Errors?

      Q8What is a readiness probe for?

      Exam-style

      Written practice

      Short answer6 marks

      Compare Blue/Green and Canary deployments in terms of risk and rollback.

      Short answer5 marks

      Which metrics would trigger an automatic rollback of a canary stroke-risk model, and why are HTTP metrics alone not enough?

      Long answer12 marks

      Plan the rollout of stroke model v2 at the referral hospital from first deployment to 100% traffic. Include the infrastructure, the stages and percentages, the gates at each stage, the rollback mechanism, and how you would compare v1 and v2 predictions safely before exposing patients.

      Watch

      Videos

      Fresh picks are chosen by AI every day.
      Hands-on

      Canary Release & Traffic Splitting Simulator

      4 guided tasks: predict, experiment, explain — then challenge yourself. Plus 1 coding exercise you run right in the browser.

      Start the lab
      Containerization, Multi-Stage Docker & Security HardeningInference Performance: ONNX, Pruning & INT8 Quantization