Data Profiling, Validation & Cleaning
DSC3103 · Lab

Data Profiler, Tukey Outlier Gate & Median Imputer

Feed a dirty market-price batch through the cleaning funnel and watch where rows go. Then see, with real numbers, why one typo drags the mean but not the median — the reason we impute per commodity with the median.

  1. 1 · Predict0/1
  2. 2 · Experiment0/3
  3. 3 · Explain0/1
  4. 4 · Code it0/2
  5. 5 · Challenge
  6. 0/5 tasks done

Predict first

Commit to a guess before you touch anything. Being wrong here is how it sticks.

5,000 rows, 6% missing keys, then 10% duplicates removed. How many clean rows?

Experiment

Move the controls and watch the outputs. Goals tick off automatically when you reach them.

The cleaning funnel

Rows lost to missing keys, then to duplicate business keys (date, market, commodity).

Raw5,000
After dropping null keys4,700
After de-duplication (clean)4,230
Retention
84.6%
Duplicates dropped
470

Tukey fences & the mean-vs-median trap

Four honest maize prices [1800, 1900, 2000, 2100] plus one value you control.

Tukey 1.5×IQR fences
[400, 5,200] UGX
IQR = 3400 − 2200 = 1200 upper = Q3 + 1.5·IQR = 3400 + 1800 = 5200
Median
2,000 UGX
Mean
3,500 UGX
Mean − median
1,500 UGX

One bad value moved the mean a lot; the median didn't budge. That's what “robust” means.

Explain it

Put what you saw into words. This is what the exam will ask you to do.

Using what you saw, explain why we impute missing prices with the per-commodity median and never edit data/raw/.

Code it

Write real code against real data. Python and SQL run right here in your browser and hidden tests check your work; the AI reviews it like an examiner.

Pythonwarm-up

Profile the batch & compute Tukey fences

Before cleaning anything, measure it. market_prices_dirty.csv is in your working directory.

  1. null_rates(df) → a dict mapping every column name to the % of missing values, rounded to 1 dp.
  2. tukey_fences(values) → a tuple (lower, upper) with Q1 − 1.5·IQR and Q3 + 1.5·IQR (use pandas' default quantiles). It must accept a list or a Series.
Data in the working directory: market_prices_dirty.csv
Ctrl/⌘ + Enter to run

Challenge

Fixed variations, a new problem every day, and unlimited AI challenges built from your current settings.

Tukey 1.5×IQR Upper Outlier Fence

In market_prices_dirty.csv, beans prices have Q1 = 2,200 UGX and Q3 = 3,400 UGX. What is the exact upper Tukey outlier fence (Q3 + 1.5 × IQR)?