Workflow Orchestration with Apache Airflow
DSC3103 ยท Lab

Airflow DAG Retry Backoff & Critical Path Simulator

Configure retries the way you would for a flaky API, then lay out a DAG with a parallel branch and find its critical path โ€” the thing that decides how long the run takes.

  1. 1 ยท Predict0/1
  2. 2 ยท Experiment0/2
  3. 3 ยท Explain0/1
  4. 4 ยท Code it0/2
  5. 5 ยท Challenge
  6. 0/4 tasks done

Predict first

Commit to a guess before you touch anything. Being wrong here is how it sticks.

retries = 4, retry_delay = 10 s, exponential backoff. Total wait if all fail?

Experiment

Move the controls and watch the outputs. Goals tick off automatically when you reach them.

Retry backoff

Total waiting if every retry fails
140 s
20 + 40 + 80 = 140 s

Critical path

ingest โ†’ (validate_schema โˆฅ profile_stats) โ†’ clean_and_store

Critical path
62 s
If run serially
70 s
Saved by parallelism
8 s

The slower branch (profile_stats) sets the pace. Speeding up the other branch saves nothing.

Explain it

Put what you saw into words. This is what the exam will ask you to do.

Why should ingest get more retries than validate, and what happens downstream when validate fails?

Code it

Write real code against real data. Python and SQL run right here in your browser and hidden tests check your work; the AI reviews it like an examiner.

Pythoncore

Retry budgets and the critical path

Two calculations you'll do in every Airflow design question:

  1. backoff_schedule(delay, retries, exponential=True) โ†’ list of waits in seconds before each retry ([20, 40, 80] for delay 20, 3 retries; [20, 20, 20] without exponential backoff).
  2. critical_path(durations, upstream) โ†’ the run time of the whole DAG, i.e. the longest path. durations maps task โ†’ seconds; upstream maps task โ†’ list of tasks it depends on.
Ctrl/โŒ˜ + Enter to run

Challenge

Fixed variations, a new problem every day, and unlimited AI challenges built from your current settings.

Airflow Exponential Backoff Total Wait

An ingest task is configured with retries=3, initial retry_delay=20 seconds, and retry_exponential_backoff=True (doubling each retry: 20s, 40s, 80s). If all retries fail, how many total seconds were spent waiting in retry delays?