Inference Performance: ONNX, Pruning & INT8 Quantization

Training is about throughput: process millions of samples per hour in big batches, and it's fine if a single step takes a second. Serving a patient is about latency: one prediction, now — typically a budget of ~10–15 ms for the model inside a ~100 ms end-to-end request.

Where inference time goes:

• Python overhead — interpreting each operation, creating objects.
• Memory traffic — moving weights from RAM to the CPU caches; often the real bottleneck.
• Arithmetic — the multiply-adds themselves.

Optimisation attacks each one: compiled runtimes remove Python overhead, smaller number formats cut memory traffic, and SIMD instructions do more arithmetic per cycle.

ONNX (Open Neural Network Exchange) is a standard file format for model computation graphs. You can export from scikit-learn (skl2onnx), XGBoost, PyTorch or TensorFlow into one .onnx file.

ONNX Runtime executes that graph with optimised C++ kernels: it fuses operations, plans memory up front and uses hardware-specific instructions — and no Python interpreter runs inside the hot loop. Typical result: the same model, same outputs, noticeably lower latency. A bonus: the .onnx file can be served from C#, Java or JavaScript, not just Python.

sess = onnxruntime.InferenceSession("stroke.onnx")
probs = sess.run(None, {"input": X.astype(np.float32)})[1]

Weights are normally 32-bit floats (FP32, 4 bytes). Quantisation stores them in fewer bits:

INT8 is 4× smaller (a 75% saving). Speed improves too: less data to move from memory, more of the model fits in cache, and CPU vector units (AVX-512 VNNI) process four times as many 8-bit values per instruction as 32-bit ones.

Affine quantisation maps a float range [min, max] onto 0–255: scale s = (max − min)/255, zero-point z = round(−min/s), q = round(x/s) + z, and dequantised x̂ = (q − z)·s. The rounding error is at most s/2 — which is why outliers that stretch the range hurt accuracy.

Pruning removes weights (or whole neurons, trees, branches) that contribute little — e.g. weights near zero. Sparse models can be smaller and faster, especially with structured pruning that removes entire channels or layers the runtime can skip.

Every optimisation is lossy or at least risky, so the rule is: re-run the full evaluation on the optimised artifact — ROC-AUC, recall at the clinical threshold, calibration — and compare with the original. In the course example INT8 + pruning cost less than 0.5% ROC-AUC, an excellent trade for a 4× smaller, several-times-faster model. If recall at the triage threshold dropped noticeably, you'd ship the FP32 model instead.