Removing Duplicates and Detecting Outliers

Duplicates & Outliers

Cleaning often begins with de‑duplication and outlier detection.

Removing duplicate rows

duplicates = df.duplicated(subset=['id'], keep='first')
print('Duplicate rows:', duplicates.sum())

df_clean = df.drop_duplicates(subset=['id'], keep='first')

Detecting outliers with Z‑score

from scipy import stats
numeric_cols = df.select_dtypes('number').columns
z_scores = np.abs(stats.zscore(df[numeric_cols]))
outliers = (z_scores > 3).any(axis=1)
print('Outlier rows:', df[outliers].index.tolist())

Removing outliers (optional)

df_no_outliers = df[~outliers]

Using IQR (Inter‑Quartile Range)

Q1 = df['price'].quantile(0.25)
Q3 = df['price'].quantile(0.75)
IQR = Q3 - Q1
filter = (df['price'] >= Q1 - 1.5*IQR) & (df['price'] <= Q3 + 1.5*IQR)
df_filtered = df[filter]

Visual check with boxplot

sns.boxplot(x=df['price'])
plt.show()