Decision Trees and Random Forests

Decision Trees & Random Forests

Tree‑based models are intuitive, handle mixed data types, and require little preprocessing.

Decision Tree basics

• Splits data based on feature thresholds.
• Uses impurity measures: Gini, entropy (information gain).

Training a Decision Tree with scikit‑learn

from sklearn.tree import DecisionTreeClassifier
from sklearn import tree

clf = DecisionTreeClassifier(max_depth=5, random_state=42)
clf.fit(X_train, y_train)

# Visualize the tree
plt.figure(figsize=(12,8))
tree.plot_tree(clf, filled=True, feature_names=X.columns, class_names=['No','Yes'])
plt.show()

Random Forests (ensemble of trees)

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=200, max_features='sqrt', random_state=42)
rf.fit(X_train, y_train)

# Feature importance
importances = pd.Series(rf.feature_importances_, index=X.columns)
importances.sort_values(ascending=False).plot(kind='bar')
plt.title('Feature Importances from Random Forest')
plt.show()

Hyperparameters to tune

• n_estimators (number of trees)
• max_depth
• min_samples_split / min_samples_leaf
• max_features

Advantages & disadvantages

	Decision Tree	Random Forest
Interpretability	High (visual)	Medium (feature importance)
Overfitting	Prone	Less prone (averaging)
Training speed	Fast	Slower (many trees)

Out‑of‑Bag (OOB) error estimate

rf.oob_score_   # gives OOB accuracy for classification