Real‑World Data Challenges: Messy Datasets

Real‑World Data Challenges

In practice, data comes from many sources and is rarely clean. Common issues include:

• Missing values (NaN, empty strings)
• Inconsistent formats (dates, currencies)
• Duplicates
• Outliers
• Mixed data types in the same column
• Large file sizes that exceed memory limits

Example: A raw CSV sample

id,date,price,category,quantity
1,2024‑01‑01,12.5,Electronics,5
2,,15.0,Clothing,3
3,2024‑01‑03,,Electronics,2
4,2024-01-04,8.75,Food,abc
5,2024‑01‑05,20.0,Electronics,7

What to look for

1. Blank fields (row 2 missing date, row 3 missing price).
2. Wrong data type (row 4 quantity is a string 'abc').
3. Inconsistent date format (row 4 uses a hyphen).

Tip: Start by loading the file into Pandas and using .info() and .describe() to spot anomalies.