Hyperparameter Tuning: Grid Search vs Random Search vs Bayesian
Grid search, random search and Bayesian optimisation compared on one real model with the same budget of 24 candidates: how close they came, why the differences sat inside the noise between folds, why random search explores more, and a second model where tuning made the difference between 61% and 93% accuracy.
GloryTecks Team
Training & Placement Faculty
Last updated: 25 September 2026
Hyperparameters are the settings you choose before training, such as the learning rate of a boosted model or the regularisation strength of an SVM. Unlike the model's weights, they are not learned from the data, so they are chosen by trying candidates and keeping the best by cross-validation. The three main search strategies are grid search, random search and Bayesian optimisation. This guide runs all three on the same model with the same budget, then shows what the results do and do not mean.
The data is the spambase dataset: 4,601 emails described by 57 frequency features, labelled spam or not (OpenML: spambase). A quarter is held out as a test set that no search ever sees, and every search uses 5-fold cross-validation on the rest.
Step 0: the defaults, and how noisy the score is
import numpy as np
import optuna
from scipy.stats import loguniform, randint
from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import (GridSearchCV, RandomizedSearchCV, StratifiedKFold,
cross_val_score, train_test_split)
data = fetch_openml(data_id=44, as_frame=True)
X, y = data.data, (data.target == "1").astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
model = HistGradientBoostingClassifier(early_stopping=False, random_state=0)
scores = cross_val_score(model, X_train, y_train, cv=cv)
print(f"defaults: mean {scores.mean():.4f}, fold scores {scores.round(3)}, spread {scores.std():.4f}")defaults: mean 0.9510, fold scores [0.952 0.948 0.948 0.962 0.945], spread 0.0061Before tuning anything, look at the spread. The five folds range from 94.5% to 96.2% accuracy with a standard deviation of 0.006. A tuned model that beats the defaults by a few thousandths may simply have been lucky with the folds. Keep that number in mind for everything that follows. Cross-validation in machine learning explains where the spread comes from.
Grid search: every combination of chosen values
Grid search tries every combination of the values you list (scikit-learn: tuning hyperparameters). Three learning rates and two values for each of three other settings make 24 candidates:
grid = {"learning_rate": [0.03, 0.1, 0.3], "max_leaf_nodes": [15, 63],
"min_samples_leaf": [10, 40], "l2_regularization": [0.0, 1.0]}
grid_search = GridSearchCV(model, grid, cv=cv).fit(X_train, y_train)
print(f"grid search, {len(grid_search.cv_results_['params'])} candidates: best CV {grid_search.best_score_:.4f}")
print(grid_search.best_params_)grid search, 24 candidates: best CV 0.9536
{'l2_regularization': 0.0, 'learning_rate': 0.3, 'max_leaf_nodes': 15, 'min_samples_leaf': 10}Grid search is simple and exhaustive within its grid, but the cost multiplies with every setting added: five values for each of four settings would be 625 candidates, each trained five times.
Random search: the same budget, more variety
Random search samples each candidate from a distribution per setting, here log-uniform for rates and strengths, and stops after a fixed number:
space = {"learning_rate": loguniform(0.01, 0.5), "max_leaf_nodes": randint(8, 128),
"min_samples_leaf": randint(5, 100), "l2_regularization": loguniform(1e-3, 10)}
random_search = RandomizedSearchCV(model, space, n_iter=24, cv=cv, random_state=0).fit(X_train, y_train)
print(f"random search, 24 candidates: best CV {random_search.best_score_:.4f}")
print({k: round(v, 3) for k, v in random_search.best_params_.items()})random search, 24 candidates: best CV 0.9530
{'l2_regularization': 0.037, 'learning_rate': 0.263, 'max_leaf_nodes': 80, 'min_samples_leaf': 14}Bergstra and Bengio showed why random search is usually the better default: for most problems only a few hyperparameters really matter, and different ones matter on different data sets (Bergstra and Bengio: random search). A grid spends its budget repeating the same few values of the important setting; random search tries a new value of every setting in every candidate. That shows up in a count after the next step.
Bayesian optimisation: learn from earlier trials
Bayesian methods build a model of how the score depends on the settings and use it to pick promising candidates. Optuna's TPE sampler is a popular implementation; its first 10 trials are random by default, and later ones are guided by the results so far (Optuna: TPESampler):
def objective(trial):
params = {"learning_rate": trial.suggest_float("learning_rate", 0.01, 0.5, log=True),
"max_leaf_nodes": trial.suggest_int("max_leaf_nodes", 8, 128),
"min_samples_leaf": trial.suggest_int("min_samples_leaf", 5, 100),
"l2_regularization": trial.suggest_float("l2_regularization", 1e-3, 10, log=True)}
return cross_val_score(model.set_params(**params), X_train, y_train, cv=cv).mean()
optuna.logging.set_verbosity(optuna.logging.WARNING)
study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=24)
print(f"Bayesian (Optuna TPE), 24 trials: best CV {study.best_value:.4f}")
print({k: round(v, 3) for k, v in study.best_params.items()})Bayesian (Optuna TPE), 24 trials: best CV 0.9525
{'learning_rate': 0.072, 'max_leaf_nodes': 117, 'min_samples_leaf': 23, 'l2_regularization': 0.001}The results, read honestly
Now the one comparison that counts: each chosen configuration trained on all the training data and scored once on the untouched test set.
chosen = {"defaults": {}, "grid": grid_search.best_params_,
"random": random_search.best_params_, "Bayesian": study.best_params}
for name, params in chosen.items():
final = HistGradientBoostingClassifier(early_stopping=False, random_state=0, **params).fit(X_train, y_train)
print(f"{name:>8}: test accuracy {final.score(X_test, y_test):.4f}")
print("distinct learning rates tried - grid:", len(set(grid_search.cv_results_["param_learning_rate"])),
"| random:", len(set(random_search.cv_results_["param_learning_rate"])))defaults: test accuracy 0.9592
grid: test accuracy 0.9592
random: test accuracy 0.9661
Bayesian: test accuracy 0.9661
distinct learning rates tried - grid: 3 | random: 24Three observations:
- All three searches beat the defaults in cross-validation, by 0.15 to 0.26 percentage points. That is less than half the fold-to-fold spread of 0.6 points. On this data, the defaults of a modern gradient-boosting library were already close to the best available.
- On the test set, random and Bayesian search score 96.6% against 95.9%. With 1,151 test emails, that gap is 8 emails, about one standard error. Suggestive, not proof.
- Random search tried 24 different learning rates to the grid's 3. That is Bergstra and Bengio's point in one line.
With only 24 trials, Optuna had just 14 guided ones after its random start, so this comparison does not show Bayesian search at its best. Its advantage grows when each trial is expensive and the budget allows the model of the score to learn, as with deep networks that take hours per trial.
When tuning really matters
Gradient boosting has good defaults. Other models do not forgive bad settings. The same random search over an SVM's two main hyperparameters, C and gamma:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
svm = make_pipeline(StandardScaler(), SVC())
print(f"SVM defaults: CV {cross_val_score(svm, X_train, y_train, cv=cv).mean():.4f}")
svm_search = RandomizedSearchCV(svm, {"svc__C": loguniform(1e-2, 1e3), "svc__gamma": loguniform(1e-4, 1)},
n_iter=24, cv=cv, random_state=0).fit(X_train, y_train)
results = svm_search.cv_results_["mean_test_score"]
print(f"SVM candidates ranged from {results.min():.4f} to {results.max():.4f}")
print({k: round(v, 4) for k, v in svm_search.best_params_.items()}, f"test accuracy {svm_search.score(X_test, y_test):.4f}")SVM defaults: CV 0.9281
SVM candidates ranged from 0.6061 to 0.9322
{'svc__C': 10.3233, 'svc__gamma': 0.0151} test accuracy 0.9470Across the candidates, cross-validated accuracy ran from 61% to 93%. A careless manual choice could land anywhere in that range; a search reliably finds the good region. Tuning matters most for models that are sensitive to their settings, and when you do not know good defaults.
Which method to use
| Situation | Method |
|---|---|
| One or two settings, cheap to train | Grid search over a sensible range |
| Three or more settings | Random search with log-uniform ranges for rates and strengths |
| Each trial is expensive (minutes to hours) | Bayesian optimisation, such as Optuna |
| Many candidates, large data | Successive halving (HalvingRandomSearchCV), which drops weak candidates early on small samples |
| Model with strong defaults | Try the defaults first; tune only if the gain exceeds the fold spread |
Rules for honest tuning
- Tune on training data with cross-validation only. The test set is used once, at the end.
- Compare gains with the spread between folds. Differences smaller than it are not evidence.
- Search rates and strengths on a log scale. The useful range of a learning rate spans orders of magnitude.
- Fix the budget in advance. Searching until a number looks good is overfitting to the validation folds.
- Report the tuned result with its spread, and for a rigorous estimate of the whole procedure, use nested cross-validation.
How many trials should a random search use?
Is Bayesian optimisation always better than random search?
Which hyperparameters matter most for gradient boosting?
Can tuning cause overfitting?
How much a tuned model is worth depends on the metric you tune for; evaluation metrics explained covers the choice. Cross-validation and hyperparameter tuning are part of the model evaluation and optimisation module of the Data Science course.
Sources
- Tuning the hyper-parameters of an estimator (scikit-learn) — scikit-learn documentation
- Random Search for Hyper-Parameter Optimization — Journal of Machine Learning Research, 2012
- optuna.samplers.TPESampler — Optuna documentation
- spambase (OpenML dataset 44) — www.openml.org
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Data Science?
Learn Data Science at GloryTecks in Hyderabad, with trainers who have industry experience, real-time projects and placement assistance.
Related articles
Data Scientist vs AI Engineer in 2027: Which Role Fits You?
Data scientists turn data into answers and predictive models; AI engineers build products with foundation models, such as retrieval systems and agents. This comparison covers the daily work, skills, tools, interviews and career paths of both roles in 2027, how certifications reflect the shift, and how to choose.
Data Science Roadmap 2027: Skills, Order and a 12-Month Plan
A data science roadmap for 2027 in four phases over twelve months: Python, pandas 3 and SQL; statistics and experiments; machine learning with honest evaluation; and LLMs, deployment and a portfolio. It covers what changed for data scientists in 2026, the tool versions to install and a month-by-month plan.
Machine Learning Skills That Still Matter in 2027, and What LLMs Changed
Language models changed how we build text applications, but most business prediction problems still run on tables, where classical machine learning is strong. These are the machine learning skills that still matter in 2027, a runnable example of why the right metric matters, and where LLMs now fit into ML work.