Skip to main content
Brave Programmer Logo

BraveProgrammer

BraveProgrammer

HomeProjectsBlogsCoursesLessonsAbout

Site footer

BraveProgrammer

Free coding courses, practical tutorials, and real projects from BraveProgrammer. Learn web development with React, Next.js, and TypeScript.

Navigation

  • Home
  • Projects
  • Blogs
  • Courses

Resources

  • About
  • Lessons
  • Teach on BraveProgrammer

© 2026 BraveProgrammer. All rights reserved.

Removing Duplicates | Master Data Science with Python lesson | BraveProgrammer
  1. Courses
  2. /
  3. Master Data Science with Python

Lesson 43 of 60

·python

Removing Duplicates and Detecting Outliers

20 minutes

Course progress72%

Duplicates & Outliers

Cleaning often begins with de‑duplication and outlier detection.

Removing duplicate rows

duplicates = df.duplicated(subset=['id'], keep='first')
print('Duplicate rows:', duplicates.sum())

df_clean = df.drop_duplicates(subset=['id'], keep=)
'first'

Detecting outliers with Z‑score

from scipy import stats
numeric_cols = df.select_dtypes('number').columns
z_scores = np.abs(stats.zscore(df[numeric_cols]))
outliers = (z_scores > 3).any(axis=1)
print('Outlier rows:', df[outliers].index.tolist())

Removing outliers (optional)

df_no_outliers = df[~outliers]

Using IQR (Inter‑Quartile Range)

Q1 = df['price'].quantile(0.25)
Q3 = df['price'].quantile(0.75)
IQR = Q3 - Q1
filter = (df['price'] >= Q1 - 1.5*IQR) & (df['price'] <= Q3 + 1.5*IQR)
df_filtered = df[filter]

Visual check with boxplot

sns.boxplot(x=df['price'])
plt.show()

Info

Be cautious when removing outliers; they may represent rare but important cases (e.g., fraud detection).
Make it stickPractice what you've learned1 quiz · 4 practical tests · Optional activities from Master Data Science with PythonBrowse activities
Previous lessonHandling Missing Values: Imputation StrategiesContinue to lesson 44Feature Scaling: Normalization and StandardizationBack to Master Data Science with Python curriculum