Dimensionality Reduction – Principal Component Analysis (PCA)

Principal Component Analysis (PCA)

PCA reduces dimensionality by projecting data onto orthogonal axes (principal components) that capture maximal variance.

Why it matters

• Visualization: reduce to 2‑D/3‑D.
• Noise reduction: discard low‑variance components.
• Speed up downstream algorithms.

Performing PCA with scikit‑learn

from sklearn.decomposition import PCA

pca = PCA(n_components=0.95)   # keep 95% variance
X_pca = pca.fit_transform(X_scaled)
print('Original shape:', X_scaled.shape)
print('Reduced shape:', X_pca.shape)

Explained variance ratio

explained = pca.explained_variance_ratio_
plt.bar(range(1, len(explained)+1), explained)
plt.xlabel('Principal Component')
plt.ylabel('Explained Variance Ratio')
plt.title('Scree Plot')
plt.show()

Visualizing first two components

plt.scatter(X_pca[:,0], X_pca[:,1], c=y, cmap='coolwarm', edgecolor='k')
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.title('PCA Projection')
plt.show()

When to use PCA

• High‑dimensional datasets (e.g., image pixels, text TF‑IDF).
• Before clustering or classification to reduce noise.

Caveats

• Linear method – cannot capture non‑linear structure.
• Components are linear combinations, not directly interpretable.