Data Cleaning: Duplicates, Renaming, and Type Conversion

Data Cleaning Essentials

Cleaning data is often 80 % of a data scientist's workload.

Removing duplicates

duplicates = df.duplicated()
print('Duplicate rows:', duplicates.sum())

df_nodup = df.drop_duplicates()

Renaming columns for clarity

df.rename(columns={'emp_id': 'employee_id', 'dept': 'department'}, inplace=True)

Changing data types

# Convert a column to datetime
df['date'] = pd.to_datetime(df['date'], format='%Y-%m-%d')

# Convert categorical column to `category` dtype for memory savings
df['department'] = df['department'].astype('category')

Stripping whitespace in string columns

df['city'] = df['city'].str.strip()

Applying a custom cleaning function

def clean_name(name):
    return name.title().strip()

df['name'] = df['name'].apply(clean_name)