Inference Performance: ONNX, Pruning & INT8 Quantization
CSC3119 · Lab

ONNX & INT8 Quantization Speedup Simulator

Shrink the model's numbers from 32 bits to 8 and move it from Python to ONNX Runtime. Watch memory, latency and throughput change — and keep an eye on accuracy.

  1. 1 · Predict0/1
  2. 2 · Experiment0/2
  3. 3 · Explain0/1
  4. 4 · Code it0/1
  5. 5 · Challenge
  6. 0/4 tasks done

Predict first

Commit to a guess before you touch anything. Being wrong here is how it sticks.

A 48 M-parameter FP32 model is quantised to INT8. How many MB of RAM are saved?

Experiment

Move the controls and watch the outputs. Goals tick off automatically when you reach them.

Precision & runtime

Weights in RAM
100 MB
Latency
30.0 ms
Speed-up vs FP32 Python
1.0×
ROC-AUC change
0
FP32100 MB
Current100 MB

Explain it

Put what you saw into words. This is what the exam will ask you to do.

How does INT8 quantisation cut memory 4× and speed up CPU inference, and what must you check afterwards?

Code it

Write real code against real data. Python and SQL run right here in your browser and hidden tests check your work; the AI reviews it like an examiner.

Pythoncore

Affine INT8 quantisation in NumPy

Implement asymmetric (affine) quantisation of a float tensor to unsigned 8-bit integers:

  • quantise(x) → (q, scale, zero_point) where scale = (max − min) / 255, zero_point = round(−min / scale), q = clip(round(x / scale) + zero_point, 0, 255) as uint8.
  • dequantise(q, scale, zero_point) → (q − zero_point) · scale as float32.

Then report the size reduction and the worst absolute error on W.

Ctrl/⌘ + Enter to run

Challenge

Fixed variations, a new problem every day, and unlimited AI challenges built from your current settings.

INT8 Model Weight Compression

An FP32 deep neural network contains 24 million parameters (4 bytes each = 96 MB). Quantized to INT8 (1 byte per parameter), what is the new weight size in MB?