DSC3103 · Module 4 of 9

Workflow Orchestration with Apache Airflow

Schedule DAGs, manage dependencies & handle retries (Lab 5)

~56 min to study properly
0%
Open lab

Cron jobs are blind to data dependencies: if ingestion takes 10 minutes longer than usual, a cron-scheduled cleaning script runs on yesterday's stale file or crashes silently.

Checkpoint · the examiner asks

In an Airflow DAG (`ingest >> validate >> clean >> store >> report`), why should retry counts and delays differ by stage, and what happens if `validate` fails?

In one minute

Airflow models a pipeline as a DAG of tasks with explicit dependencies, so a failed upstream task marks downstream tasks upstream_failed and bad data never reaches storage. Give network-facing ingestion several retries with exponential backoff; let deterministic validation fail fast. Because of retries and backfills, every task must be idempotent.

Traps examiners look for
  • Forgetting a dependency edge (e.g. stored >> report_gen) and creating a race condition.
  • Retrying deterministic schema failures — they won't fix themselves.
  • Passing whole DataFrames through XCom instead of small metadata like paths.
Read

Study notes

01Why cron isn't orchestration

cron runs a command at a time. It has no idea whether yesterday's ingest finished, whether the input file exists, or whether the previous step failed. If ingest takes 10 minutes longer than usual, the 06:00 cleaning job happily processes yesterday's file — or crashes — and nobody is told.

An orchestrator like Apache Airflow manages dependencies rather than just times: "run clean only after validate succeeds". It also gives you retries, alerts, logs per task, a UI showing every run, and the ability to rerun history.

02DAGs, tasks and states

An Airflow pipeline is a DAG — a Directed Acyclic Graph:

  • Directed: edges have a direction (ingest >> validate means validate runs after ingest).
  • Acyclic: no loops — a task can't (even indirectly) depend on itself, so there is always a valid order to run things.

Tasks are created with operators or the TaskFlow @task decorator. Each task instance moves through states: scheduled → queued → running → success or failed. When a task fails, every task downstream of it becomes upstream_failed and does not run. That's the data-quality guarantee: if validate rejects the batch, clean, store and report never touch it.

ingest() >> validated() >> cleaned() >> stored() >> report_gen()

Forgetting an edge is a real bug: without stored >> report_gen, the report can run in parallel with the load and read stale or empty tables (the Lab 5 bug).

03Retries, backoff and the critical path

Failures come in two kinds:

  • Transient — network timeouts, a busy API, a locked file. Waiting and retrying usually works. Give these tasks retries=3 with exponential backoff: wait d, then 2d, then 4d… Total wait for R retries = d × (2^R − 1).
  • Deterministic — a missing column, a broken schema, a bug. Retrying produces the same failure, wastes time and delays the alert. Let these fail fast.

When tasks can run in parallel (e.g. validate_schema and profile_stats both after ingest), the run's duration is set by the critical path: the longest chain through the DAG. Speeding up a task that isn't on the critical path makes the run no faster.

04Backfills, XCom and idempotent tasks

Every DAG run has a logical date (data_interval_start) — the period of data it is responsible for, not the time it actually runs. With catchup=True, Airflow creates runs for every missed interval — a backfill. Tasks should use the logical date to decide which data to process (ds in templates), so rerunning "2025-03-04" always processes March 4th.

XCom passes small metadata between tasks: a file path, a row count, a batch id. Don't push whole DataFrames through XCom — write data to storage and pass the path.

Because of retries and backfills, every task must be idempotent: rerunning a task for the same logical date must produce the same result (overwrite that date's partition or upsert, never blind append).

Watch it done

Worked examples

Worked example 1

Exponential backoff budget

ingest has retries = 4, retry_delay = 15 s and exponential backoff. If every attempt fails, how long does Airflow spend waiting between attempts?

    Worked example 2

    Finding the critical path

    ingest (10 s) → [validate_schema (6 s) ∥ profile_stats (20 s)] → clean (15 s) → store (5 s). How long does the run take, and how much faster would it be if validate_schema took 2 s?

      Vocabulary

      Key terms

      Never pass gigabytes through Airflow XComs

      Airflow stores XCom values in its metadata database. Passing a 500 MB DataFrame in XCom will choke the scheduler DB. Always write data to object storage/Parquet and pass only the URI string and row count in XCom!

      Practise

      Quick check

      0 of 8 correct so far. Wrong answers come with an explanation — and the buddy can unpack them.

      Q1In `ingest >> validated >> cleaned >> stored >> report_gen`, if `stored` is NOT wired before `report_gen` (`stored >> report_gen` omitted), what bug occurs?

      Q2A task has initial retry_delay = 30s and exponential backoff (×2 each retry). If all 3 retries fail, what is the total wait time spent in retry delays?

      Q3What does 'acyclic' guarantee in a DAG?

      Q4validate fails. What state do clean, store and report get?

      Q5Which task should get the most retries?

      Q6What should be passed through XCom?

      Q7With catchup=True and a DAG paused for 5 days, what happens when you unpause it (daily schedule)?

      Q8retries = 3, retry_delay = 20 s, exponential. Total wait if all fail?

      Exam-style

      Written practice

      Short answer6 marks

      Explain why ingestion tasks should have more retries than validation tasks, and what Airflow does downstream when validation fails.

      Short answer5 marks

      What is a backfill, and why must Airflow tasks be idempotent?

      Long answer12 marks

      Design an Airflow DAG for the daily market-price pipeline (ingest, validate, profile, clean, store, report). Draw the dependencies, choose retry settings for each task with reasons, identify the critical path given ingest 12 s, validate 8 s, profile 25 s, clean 15 s, store 10 s, report 5 s, and explain how you make the DAG safe to backfill.

      Watch

      Videos

      Fresh picks are chosen by AI every day.
      Hands-on

      Airflow DAG Retry Backoff & Critical Path Simulator

      4 guided tasks: predict, experiment, explain — then challenge yourself. Plus 2 coding exercises you run right in the browser.

      Start the lab
      Analytical Star Schemas & DuckDBDistributed Batch Processing with PySpark