Distributed Batch Processing with PySpark
DSC3103 · Lab

PySpark Partition Waves, Shuffle & Cache Break-Even

See how Spark turns partitions into waves of tasks, how much data a groupBy shuffles, and the exact point where .cache() stops costing you and starts paying off.

  1. 1 · Predict0/1
  2. 2 · Experiment0/2
  3. 3 · Explain0/1
  4. 4 · Code it0/2
  5. 5 · Challenge
  6. 0/4 tasks done

Predict first

Commit to a guess before you touch anything. Being wrong here is how it sticks.

20 partitions on 8 cores, 4 s per task. How long does the stage take?

Experiment

Move the controls and watch the outputs. Goals tick off automatically when you reach them.

Partitions, cores & waves

Waves
3
Stage time
12 s
Idle core-slots in last wave
4
Max rows shuffled (partial agg)
120

Partial aggregation: each partition pre-aggregates, so at most partitions × keys rows cross the network in the shuffle.

When does .cache() pay off?

Without cache6.0 s
With cache8.0 s

Caching is slower at 2 actions — exactly what we measured in Lab 8 (8.85 s cached vs 3.04 s).

Explain it

Put what you saw into words. This is what the exam will ask you to do.

Narrow vs wide transformations — and why DuckDB beat Spark and .cache() slowed things down on our 10,000-row dataset.

Code it

Write real code against real data. Python and SQL run right here in your browser and hidden tests check your work; the AI reviews it like an examiner.

Pythoncore

Partial aggregation, by hand

Spark computes groupBy("commodity").avg("price_ugx") in two steps so it shuffles little data. partitions is a list of 4 DataFrames.

  1. partial(part) → for one partition, a DataFrame with columns commodity, total, n (sum and count of price_ugx per commodity) — this is what each executor sends in the shuffle.
  2. combine(partials) → final DataFrame with commodity, avg_price, sorted by commodity.
  3. Print how many rows were shuffled (the total length of all partials) vs rows in full.
Data in the working directory: market_clean.csv
Ctrl/⌘ + Enter to run

Challenge

Fixed variations, a new problem every day, and unlimited AI challenges built from your current settings.

Executor Core Waves Calculation

A PySpark stage has 20 partitions (each taking 4 seconds) running on a cluster with 8 executor cores. Assuming uniform partition sizes, how many seconds does the stage take to complete?