Streaming, Event-Time Windows & Watermarks
DSC3103 · Lab

Event-Time Tumbling Window & Watermark Simulator

Play the stream consumer: decide which window an event lands in, or whether it's already too late — then account for every event that came in.

  1. 1 · Predict0/1
  2. 2 · Experiment0/2
  3. 3 · Explain0/1
  4. 4 · Code it0/1
  5. 5 · Challenge
  6. 0/4 tasks done

Predict first

Commit to a guess before you touch anything. Being wrong here is how it sticks.

max_ts = 1,240 s and L = 5 s. What is the watermark — and is an event at 1,233 s on time?

Experiment

Move the controls and watch the outputs. Goals tick off automatically when you reach them.

Watermark router

Decision
ACCEPT → window [1230, 1240)
W(t) = max_ts − L = 1240 − 5 = 1235 s event_ts 1238 ≥ 1235

Windows ending at or before 1235 s are final and have been emitted.

Stream accounting

Windowed (valid)
220
Dropped duplicates
18
Late side-output
12

Explain it

Put what you saw into words. This is what the exam will ask you to do.

Explain event time vs processing time and how a watermark decides when a window is final.

Code it

Write real code against real data. Python and SQL run right here in your browser and hidden tests check your work; the AI reviews it like an examiner.

Pythonexam

A watermarking stream consumer

Implement consume(events, lateness=5, size=10) processing events in arrival order:

  1. If the event's id was already seen → record it in duplicates and skip it.
  2. Update the max event time seen; watermark W = max_ts − lateness. If ts < W → record the id in late and skip it.
  3. Otherwise add the price to its tumbling window, keyed by window start floor(ts / size) * size.

Return {"windows": {start: [prices...]}, "late": [ids], "duplicates": [ids]}. Check the duplicate before the watermark. (Order of the watermark update matters — update it before checking lateness.)

Ctrl/⌘ + Enter to run

Challenge

Fixed variations, a new problem every day, and unlimited AI challenges built from your current settings.

Watermark Threshold & Late Event Routing

A consumer tracks a watermark with allowed lateness L = 5s. The maximum seen event_ts so far is 1,240s. What is the current watermark timestamp W(t)?