API Engineering with FastAPI, Pydantic & Async Workers

A model file on disk can't help a nurse at triage. Wrapping it in a REST API lets any system — the hospital's electronic health record, a mobile app, a dashboard — send patient data over HTTP and get a prediction back as JSON.

FastAPI is the de-facto choice for Python ML serving: it's built on Starlette (async web) and Pydantic (validation), generates interactive OpenAPI docs at /docs automatically, and is fast enough that the model, not the framework, is the bottleneck.

class Patient(BaseModel):
    age: float = Field(ge=0, le=120)
    avg_glucose_level: float = Field(gt=0, le=400)
    bmi: float | None = Field(default=None, ge=10, le=80)
    hypertension: bool
    smoking_status: Literal["never", "formerly", "smokes", "unknown"]

@app.post("/predict")
def predict(p: Patient) -> Prediction: ...

Every request is checked at the boundary: wrong types, missing required fields, out-of-range values and unknown categories are rejected with HTTP 422 Unprocessable Entity and a JSON body naming the exact field and problem. Your model code never sees invalid input, so it can't crash or — worse — return a prediction for an impossible patient.

Validation is also documentation: the same schema appears in /docs for the integrators.

async def endpoints let one process handle many requests that are waiting — on a database, a log write, another HTTP call. But model inference is CPU-bound: while XGBoost computes, the event loop is blocked and nothing else is served.

Options:

• Declare the endpoint with plain def: FastAPI runs it in a threadpool.
• Run several Uvicorn workers (processes): uvicorn app.main:app --workers 4 or Gunicorn with Uvicorn workers. Each worker loads its own model copy.
• Offload heavy jobs to a queue (Celery/RQ) and return a job id.

Load the model once per worker at startup, never per request.

Little's Law: L = λ × W — average requests in the system = arrival rate × average time each spends. At 120 req/s and 25 ms latency, 3 requests are in flight on average.

Capacity: with c workers and service time S per request, the service can complete c / S requests per second (4 workers × 1000/20 ms = 200 req/s). Utilisation ρ = λ / capacity.

Latency doesn't grow linearly with load. A simple queueing approximation is W ≈ S / (1 − ρ): at ρ = 50% latency doubles; at 90% it's 10×; at ρ ≥ 1 the queue grows without bound. That's why services are sized to run at roughly 60–70% utilisation, and why p99 latency (the slowest 1%) is monitored, not just the mean.