CSC3119 · Revision guide

AI Model Deployment and Scaling on a few pages

The night-before sheet: every module's one-minute summary, traps, terms, formulas and the answer examiners want.

The 3-beat answer

Examiners reward structure. Every answer, about 30 seconds:

1 · ClaimOne sentence: what it is or does. State the rule directly.

2 · MechanismHow it works: the maths, the steps, the data structures.

3 · Trade-offWhen it fails, what it costs, or why the alternative breaks.

Formulas to know cold

Little's Law (API Concurrency): L = λ × W (concurrency = arrival rate × latency)
INT8 Quantization Memory: Size(INT8) = Size(FP32) / 4 = 0.25 × Size(FP32)
Population Stability Index (PSI): PSI = Σ (A_i − E_i) × ln(A_i / E_i)
Clinical Recall (Sensitivity): Recall = TP / (TP + FN) (proportion of strokes flagged)
Precision (Positive Predictive Value): Precision = TP / (TP + FP) (proportion of alerts true)
F-Beta Score (Clinical Emphasis on Recall): F_β = (1 + β²) × (Precision × Recall) / ((β² × Precision) + Recall)
Quantization Affine Mapping: q = round(r / S) + Z (scale S, zero-point Z)
Tukey IQR Outlier Fences: [Q1 − 1.5×IQR, Q3 + 1.5×IQR] (IQR = Q3 − Q1)
Error Budget Depletion Rate: Burn Rate = Observed Error Rate / Allowable Error Rate
Spot Instance Cost Savings: Savings% = (1 − Spot Price / On-Demand Price) × 100
SHAP Additive Efficiency Property: f(x) = φ_0 + Σ φ_i (base risk + feature attributions)
Amdahl's Law for Inference Parallelism: Speedup = 1 / ((1 − p) + p / s)
Cross-Entropy Loss with Focal Weighting: FL(p_t) = −α_t (1 − p_t)^γ ln(p_t)

Module 1

Data Preparation, Preprocessing Leaks & Clinical Imbalance

Leakage is the cardinal sin: fit scalers, imputers and SMOTE on the training split only, ideally inside an sklearn Pipeline/ColumnTransformer, or test information inflates your metrics. With ~4.9% stroke prevalence, a model that always predicts 'no stroke' scores 95% accuracy and misses every patient — judge clinical models by recall, precision and PR-AUC.

Calling fit_transform on the full dataset before train_test_split.
Applying SMOTE to the test set.
Celebrating 95% accuracy on a 5%-positive problem.

Data Leakage: Accidentally sharing information from outside the training dataset (e.g. test set statistics) when building a model.
Class Imbalance: Severe disparity in outcome classes (e.g. 95% non-stroke vs 5% stroke in clinical screening).
SMOTE: Synthetic Minority Over-sampling Technique: creates synthetic interpolated examples between k-nearest minority neighbors.
ColumnTransformer: An scikit-learn estimator that executes distinct preprocessing pipelines (numeric scalers, categorical encoders) on specific column subsets.
Recall (Sensitivity): True Positives divided by (True Positives + False Negatives); measures the proportion of actual clinical stroke cases caught.

Likely question: Why is accuracy a dangerous metric for clinical triage with 5% positive cases, and why must data scalers and SMOTE oversampling never touch the test set?

With a 5% positive stroke prevalence, a naive model that predicts negative for every patient achieves 95.1% accuracy but has zero clinical utility because Recall is 0%. In medical triage, False Negatives can be fatal, so we prioritize Recall and PR-AUC over raw accuracy. Scalers and SMOTE must only touch the training split: fitting scalers on the full dataset leaks test distribution parameters (mean and variance) into the model, and applying SMOTE across test data creates synthetic evaluation targets, destroying the statistical independence required for valid real-world generalization.

Module 2

Model Persistence: Pickle vs Joblib & Pipeline Packaging

Serialisation turns a trained model into bytes on disk. Joblib beats pickle for NumPy-heavy models (memory-mapping, compression). Always save the whole Pipeline — preprocessing plus model — or the API receives unscaled, unencoded inputs. Pickle/joblib can execute arbitrary code on load, so only load files from trusted, checksummed sources.

Saving only the classifier and re-implementing preprocessing in the API.
Loading a .pkl from an unknown URL.
Saving with one library version and loading with another without pinning.

Serialization: Converting in-memory Python object graphs into a persistent byte sequence (file or stream).
Joblib: SciPy-optimized serialization library tailored for large NumPy ndarrays and scikit-learn models using memory-mapping.
Train-Serve Skew: Discrepancy between how features are computed during training versus real-time serving.
Arbitrary Code Execution: A critical security flaw where unpickling unvalidated binary data executes attacker-controlled system calls.
End-to-End Pipeline: A composite object linking Imputer → Scaler → OneHotEncoder → Classifier into a single serializable artifact.

Likely question: Explain why saving an isolated model estimator without its preprocessing pipeline breaks production deployments, and describe the security vulnerability inherent in Python pickle deserialization.

Saving an isolated estimator without its preprocessing pipeline introduces train-serve skew. Raw API payloads cannot be ingested directly: numerical features must be normalized by the exact training scaler parameters, and categorical features must be transformed using the exact one-hot encoder categories. Bundling everything into an end-to-end `Pipeline` object guarantees identical preprocessing. Regarding security, Python's pickle format is not secure: deserialization executes arbitrary constructor logic defined in `__reduce__`, allowing an attacker to run malicious shell commands or reverse shells simply by having your server call `pickle.load()` on an untrusted file.

Module 3

API Engineering with FastAPI, Pydantic & Async Workers

Wrap the model in a FastAPI service so other systems can call it over HTTP. Pydantic validates every field at the boundary (age 0–120, positive glucose, enum categories) and returns HTTP 422 instead of crashing the model. Use async for I/O-bound work and multiple Uvicorn workers or threadpools for CPU-bound inference; Little's Law L = λ·W links throughput, latency and concurrency.

Doing CPU-heavy inference directly in an async handler and blocking the event loop.
Validating inputs inside model code instead of at the API boundary.
Forgetting to convert milliseconds to seconds in Little's Law.

FastAPI: High-performance, modern Python web framework for building APIs with Starlette and Pydantic.
Pydantic BaseModel: Data validation and settings management class using Python type annotations to validate payloads.
HTTP 422: Unprocessable Entity: client status code indicating the request body was well-formed JSON but failed semantic schema validation.
Uvicorn: Lightning-fast ASGI web server implementation for Python, running asynchronous event loops via uvloop.
Throughput (QPS): Queries Per Second: the volume of inference requests successfully completed by the server per unit of time.

Likely question: How does Pydantic schema validation enforce data contracts at the API boundary, and why is this critical for maintaining 99.9% uptime in production healthcare ML?

Pydantic enforces strict data contracts at the network boundary before payloads ever reach model logic. It verifies data types (converting string numbers to floats where valid), rejects malformed keys, and validates domain rules (e.g. `Field(..., ge=0, le=125)` for age, or allowed smoking enums). If invalid data arrives, FastAPI instantly returns a structured HTTP 422 response with exact field-level error messages. In production healthcare ML, this prevents unhandled Python exceptions (like ValueError in scikit-learn), avoids silent NaN corruption, shields backend logs from crashing, and guarantees 99.9% service reliability.

Module 4

Cloud Computing Concepts, Service Models & Hardware Sizing

Cloud gives elastic, on-demand compute. Know NIST's five characteristics and the service models: IaaS (VMs), PaaS (managed platforms like Render), SaaS, MLaaS (SageMaker, Vertex AI). Tabular models run cheaply on CPUs; large neural nets need GPUs that should scale to zero when idle. Spot/preemptible VMs are 70–90% cheaper but can be reclaimed, so use them for checkpointed training and batch jobs.

Serving a small XGBoost model on an always-on GPU.
Running latency-critical serving on Spot instances without fallback.
Confusing PaaS with IaaS — who manages the OS?

NIST Cloud Characteristics: The 5 foundational pillars: On-demand self-service, Broad network access, Resource pooling, Rapid elasticity, Measured service.
IaaS: Infrastructure as a Service: rent virtual machines, raw storage, and virtual networks.
PaaS: Platform as a Service: deploy containerized web services without managing underlying operating system updates.
MLaaS: Machine Learning as a Service: end-to-end cloud platforms dedicated to ML training, registries, and autoscaling inference.
Spot / Preemptible VM: Discounted cloud instances that the provider may terminate with short notice when capacity is needed elsewhere.

Likely question: Differentiate IaaS, PaaS, and MLaaS in the context of deploying an ML model, and explain how Spot instances reduce cloud costs while managing termination risk.

IaaS (Infrastructure as a Service) provides raw VMs where you manage the OS, CUDA drivers, Docker, and networking. PaaS (Platform as a Service, e.g. Render) abstracts the VM: you provide code/Docker and the platform manages uptime and SSL. MLaaS (Machine Learning as a Service, e.g. AWS SageMaker) provides managed model endpoints, auto-scaling, model registries, and drift monitoring. Spot (Preemptible) instances offer surplus cloud capacity at a 70–90% discount. To mitigate termination risk, workloads must be stateless or save model checkpoints frequently to durable object storage (S3/GCS), allowing jobs to resume automatically when new capacity is provisioned.

Module 5

Containerization, Multi-Stage Docker & Security Hardening

Docker packages code, runtime and libraries into immutable layered images so 'works on my laptop' works everywhere. Multi-stage builds compile wheels in a heavy builder stage and copy only the results into a slim base, cutting size by ~80% and removing compilers from the attack surface. Run as a non-root user and use .dockerignore to keep secrets and data out of the image.

Shipping gcc and build tools in the production image.
Running the container as root.
Copying source before installing requirements, which breaks layer caching.

Docker Image Layer: A read-only filesystem diff created by instructions (RUN, COPY, ADD) in a Dockerfile, cached for fast rebuilds.
Multi-Stage Build: A Docker feature allowing developers to use multiple FROM statements, copying only necessary artifacts between stages.
Base Image: The foundational container image (e.g. python:3.11-slim) providing OS userland and runtime dependencies.
Rootless Container: Running the main container application process as an unprivileged user (UID >= 1000) rather than root.
.dockerignore: Configuration file specifying files and directories to omit when sending the build context to the Docker daemon.

Likely question: Explain the architecture of a multi-stage Docker build for a Python ML serving image, and detail two security best practices for production container deployment.

A multi-stage Docker build splits compilation from execution. The first stage (`FROM python:3.11 AS builder`) installs build tools (`gcc`, `g++`, `libpq-dev`) to compile heavy Python dependencies and generate wheels. The final production stage (`FROM python:3.11-slim`) copies only the pre-compiled wheels and application code, discarding compiler binaries, header files, and build cache. This shrinks image size from ~1.2GB down to ~160MB, accelerating container startup and scaling. Two critical security best practices are: 1) Enforcing a dedicated non-root user (`RUN useradd -m appuser && USER appuser`) to block host privilege escalation if a vulnerability is exploited; and 2) Using a strict `.dockerignore` file to exclude virtualenvs, test datasets, git history, and secrets.

Module 6

Serving Architectures, Load Balancing & Canary Traffic Routing

Zero-downtime releases put a load balancer in front of healthy replicas. Blue/Green flips all traffic from v1 to a verified v2 at once; Canary sends a small slice (e.g. 5%) to v2 and watches error rate, p99 latency and prediction anomalies, rolling back automatically when the error budget is breached. Canary limits the blast radius of a bad model.

Thinking Blue/Green limits blast radius — it switches 100% at once.
Rolling out a canary without automated rollback thresholds.
Watching only HTTP errors and missing silent prediction anomalies.

Canary Deployment: Routing a small percentage of live production traffic to a new version to validate performance before wide rollout.
Blue/Green Deployment: Maintaining two identical environments, flipping router traffic entirely from the active to the standby environment.
Blast Radius: The scope and severity of damage or user impact if a new deployment experiences bugs or downtime.
Readiness Probe: A container check ensuring the application has finished initializing (e.g. loaded model weights) before receiving traffic.
Error Budget: The maximum allowable threshold of errors or degraded latency permitted before deployment gates halt and roll back.

Likely question: Compare Blue/Green and Canary deployment strategies for machine learning models. How do you configure health checks and error budgets to automate rollbacks?

Blue/Green deployment maintains two identical full environments; once Green is validated, router traffic flips 100% all at once. While fast, if a subtle bug or edge-case regression exists, 100% of incoming users are affected immediately. In contrast, Canary deployment routes a tiny initial fraction (e.g. 5% to 10%) of live traffic to the new model (v2) while 90% stays on stable v1, drastically minimizing the blast radius. Health checks use readiness probes (checking if model weights are loaded in memory) and liveness probes (verifying the HTTP loop responds). Automated rollbacks monitor an error budget: if the Canary's HTTP 5xx rate exceeds 1% or p99 latency spikes above 250ms over a 2-minute rolling window, the ingress controller shifts 100% of traffic back to v1 automatically.

Module 7

Inference Performance: ONNX, Pruning & INT8 Quantization

Inference is about single-request latency. ONNX Runtime compiles the model graph to optimised C++ kernels, removing Python overhead. INT8 quantisation stores 1 byte per weight instead of 4 (75% smaller) and SIMD/VNNI instructions process four times more INT8 values per cycle. Combined with pruning, you get big speedups for a tiny accuracy cost — always re-measure AUC after.

Saying INT8 saves 25% — it saves 75% (4× smaller).
Quantising without re-validating clinical metrics.
Optimising training throughput when the problem is inference latency.

FP32: 32-bit single-precision floating point format; standard default for training ML models.
INT8 Quantization: Converting 32-bit floats to 8-bit signed/unsigned integers to reduce memory bandwidth and accelerate arithmetic.
ONNX Runtime: A cross-platform, high-performance scoring engine for Open Neural Network Exchange models.
Weight Pruning: Removing redundant parameters or decision branches with near-zero weights to sparsify models.
SIMD / VNNI: Single Instruction Multiple Data / Vector Neural Network Instructions: hardware-level parallelism for ultra-fast integer matrix dot products.

Likely question: Explain the mathematical principle of post-training quantization from FP32 to INT8. Why does it shrink memory by 4x, and what hardware acceleration does INT8 leverage?

Post-training quantization maps continuous 32-bit floating point values (FP32) into discrete 8-bit integers (INT8) via linear affine transformation: $q = \text{round}(r / S) + Z$, where $S$ is a scale factor and $Z$ is the zero-point offset. Because FP32 requires 4 bytes per parameter while INT8 requires only 1 byte, the model's disk and RAM memory footprint is reduced by exactly 4× (75% savings). In production inference, memory bandwidth is often the primary bottleneck; transferring 4× fewer bytes from RAM to CPU cache significantly speeds up execution. Additionally, modern CPUs utilize SIMD vector extensions (Intel AVX-512 VNNI, ARM NEON) that execute multiple INT8 multiply-accumulate operations in a single clock cycle, often yielding a 2× to 4× reduction in inference latency with negligible degradation in accuracy.

Module 8

Observability, Covariate Drift & Population Stability Index (PSI)

ML fails silently: HTTP 200, wrong answers. Data drift is a change in P(X); concept drift is a change in P(Y|X). PSI bins a feature and sums (A − E)·ln(A/E) across bins: < 0.1 stable, 0.1–0.25 monitor, ≥ 0.25 retrain and recalibrate. Monitor inputs and outputs continuously, not just uptime.

Using log base 10 instead of the natural log in PSI.
Treating a PSI of 0.18 as 'retrain now' — it's the monitor band.
Confusing data drift with concept drift.

Data Drift (Covariate Shift): Change in the marginal distribution of input features P(X) between training and production inference.
Concept Drift: Change in the conditional distribution P(Y|X); the relationship between inputs and ground-truth targets fundamentally changes.
Population Stability Index (PSI): A symmetric metric derived from Kullback-Leibler divergence measuring the shift between two probability distributions.
Kolmogorov-Smirnov (KS) Test: A non-parametric statistical hypothesis test comparing continuous cumulative distributions to detect significant divergence.
Automated Retraining Trigger: A monitored workflow (e.g. in Airflow) that automatically initiates model training when drift or performance metrics breach thresholds.

Likely question: Define Data Drift versus Concept Drift with clinical examples, and explain how the Population Stability Index (PSI) is calculated and interpreted.

Data Drift (covariate shift) occurs when the input feature distribution $P(X)$ changes over time while the true relationship remains the same—e.g., if hospital outreach causes the proportion of elderly patients (age > 65) to double from training baseline. Concept Drift occurs when the underlying statistical relationship between features and target $P(Y|X)$ changes—e.g., if a new preventative hypertension medication decouples high blood pressure from subsequent stroke incidence. The Population Stability Index (PSI) quantifies covariate shift by partitioning features into reference bins (quantiles) and calculating: $\text{PSI} = \sum_{i=1}^k (A_i - E_i) \times \ln(A_i / E_i)$, where $E_i$ is expected baseline frequency and $A_i$ is actual production frequency. Standard industry interpretation: PSI < 0.1 means no significant change; 0.1 ≤ PSI < 0.25 indicates moderate drift; and PSI ≥ 0.25 indicates severe drift, automatically alerting engineering and triggering retraining pipelines.

Module 9

Capstone: Regional Referral Hospital Stroke Risk Pipeline & Explainable AI (XAI)

The capstone ties everything together for the Regional Referral Hospital: with ~5% prevalence the default 0.5 threshold misses most strokes, so calibrate it (e.g. 0.22) to maximise recall within triage capacity. Physicians need explanations, not bare probabilities: SHAP gives each feature's additive contribution from the base rate to this patient's risk, turning a score into clinical action.

Keeping the 0.5 threshold on a 5%-positive problem.
Presenting a risk score with no explanation to a physician.
Reading SHAP values as causal effects rather than model attributions.

Threshold Calibration: Adjusting the classification decision boundary away from 0.5 to optimize clinical utility under asymmetric error costs.
Explainable AI (XAI): Methods and techniques in machine learning that make the outputs and internal reasoning of models interpretable to humans.
SHAP (SHapley Additive exPlanations): Game-theoretic approach to explain individual predictions by computing the average marginal contribution of each feature across all subsets.
Local Feature Attribution: Quantifying how much each specific input feature value shifted the prediction for an individual patient relative to the dataset average.
Chronic Clinic Integration: Uganda Ministry of Health June 2025 protocol integrating predictive risk triage to refer high-risk cardiovascular patients into longitudinal care.

Likely question: Why must clinical classification thresholds be calibrated away from the default 0.5, and how does SHAP (Shapley Additive exPlanations) establish clinical trust for attending doctors?

The default 0.5 probability threshold assumes symmetric misclassification costs (a False Positive equals a False Negative). In clinical triage, a False Negative (missing a high-risk stroke patient who then suffers permanent disability or death) is catastrophic, whereas a False Positive (an extra diagnostic evaluation or chronic clinic check) is low-cost. We calibrate the decision threshold down (e.g. to 0.20–0.25) to achieve 85%+ Recall. SHAP values, rooted in cooperative game theory, establish clinical trust by decomposing the final prediction into exact, additive feature attributions: $\text{Prediction} = \text{Base Risk} + \sum \phi_i$. Rather than seeing an untrusted black-box probability, the doctor sees precisely which risk factors (e.g., Age +16%, Hypertension +14%, Glucose +8%) drove the prediction, validating clinical plausibility and guiding immediate therapeutic intervention.