Hyperparameter Tuning – Grid Search & Randomized Search

Hyperparameter Tuning

Choosing the right hyperparameters often makes the difference between a mediocre and a state‑of‑the‑art model.

Grid Search (exhaustive)

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {
    'n_estimators': [100, 200, 300],
    'max_depth': [None, 10, 20],
    'min_samples_split': [2, 5, 10]
}

grid = GridSearchCV(RandomForestClassifier(random_state=42),
                    param_grid,
                    cv=5,
                    scoring='f1',
                    n_jobs=-1)
grid.fit(X_train, y_train)
print('Best params:', grid.best_params_)
print('Best CV F1:', grid.best_score_)

Randomized Search (sampled)

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_dist = {
    'n_estimators': randint(100, 500),
    'max_depth': randint(5, 50),
    'min_samples_leaf': randint(1, 10)
}

rand_search = RandomizedSearchCV(RandomForestClassifier(random_state=42),
                                 param_dist,
                                 n_iter=30,
                                 cv=5,
                                 scoring='roc_auc',
                                 random_state=42,
                                 n_jobs=-1)
rand_search.fit(X_train, y_train)
print('Best params (random):', rand_search.best_params_)

Bayesian Optimization (advanced, optional) – using scikit‑optimize

pip install scikit-optimize

from skopt import BayesSearchCV

opt = BayesSearchCV(RandomForestClassifier(random_state=42),
                     {"n_estimators": (100, 500), "max_depth": (5, 50)},
                     n_iter=32,
                     cv=5,
                     scoring='accuracy',
                     n_jobs=-1,
                     random_state=0)
opt.fit(X_train, y_train)
print('Best params (bayes):', opt.best_params_)

Practical tips

• Scale hyperparameter space (log scale for learning rates).
• Use a validation set distinct from cross‑validation if you need to tune many parameters.
• Avoid overfitting to the CV score – keep a final hold‑out test.

Early stopping (for gradient boosting/NN)

from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=1000, learning_rate=0.01, validation_fraction=0.1, n_iter_no_change=10)
model.fit(X_train, y_train)