Handling Imbalanced Datasets: Techniques That Work
Imbalanced dataset techniques compared on real screening data where only 2.3% of cases are positive: what class weights, oversampling, SMOTE and undersampling really change, why moving the threshold did better, how a stronger model beat every resampling trick, and how resampling before the split fakes a great score.
GloryTecks Team
Training & Placement Faculty
Last updated: 25 September 2026
When one class is rare, a classifier can score well by mostly ignoring it. Fraud, equipment failures, rare diseases and customer churn all look like this. The usual advice is to rebalance the data with oversampling, undersampling, SMOTE or class weights. Those techniques do something real, but it is often not what people expect, and there are simpler options that can work better.
This guide measures the techniques on a real problem: 11,183 mammography cases described by six numeric features, of which only 260, or 2.3%, are positive (OpenML: mammography). Every result uses 5-fold cross-validated predictions, so each case is scored by a model that never saw it.
The data and the evaluation
import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_recall_curve, precision_score, recall_score
from sklearn.model_selection import StratifiedKFold, cross_val_predict
from sklearn.preprocessing import StandardScaler
from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.pipeline import make_pipeline
from imblearn.under_sampling import RandomUnderSampler
data = fetch_openml(data_id=310, as_frame=True)
X, y = data.data.to_numpy(float), (data.target == "1").astype(int).to_numpy()
print(X.shape, "| positives:", y.sum(), f"({y.mean():.1%})")(11183, 6) | positives: 260 (2.3%)Two measurements are reported for each model. Average precision (AP) summarises the whole precision-recall curve: how well the model ranks positive cases above negative ones, independent of any threshold. Then, at the default threshold of 0.5, precision (the share of flagged cases that are real), recall (the share of real cases flagged) and the number flagged. Accuracy is useless here: predicting "negative" for everyone scores 97.7%. Evaluation metrics explained covers why.
What class weights and resampling actually change
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
def evaluate(model):
proba = cross_val_predict(model, X, y, cv=cv, method="predict_proba")[:, 1]
flagged = proba >= 0.5
print(f"AP {average_precision_score(y, proba):.3f} | at 0.5: precision {precision_score(y, flagged, zero_division=0):.3f}, "
f"recall {recall_score(y, flagged):.3f}, flagged {flagged.sum()}")
return proba
def logistic(**options):
return LogisticRegression(max_iter=1000, **options)
techniques = {
"no treatment": make_pipeline(StandardScaler(), logistic()),
"class weights": make_pipeline(StandardScaler(), logistic(class_weight="balanced")),
"oversampling": make_pipeline(StandardScaler(), RandomOverSampler(random_state=0), logistic()),
"SMOTE": make_pipeline(StandardScaler(), SMOTE(random_state=0), logistic()),
"undersampling": make_pipeline(StandardScaler(), RandomUnderSampler(random_state=0), logistic()),
}
probas = {}
for name, model in techniques.items():
print(f"{name:>13}:", end=" ")
probas[name] = evaluate(model) no treatment: AP 0.615 | at 0.5: precision 0.784, recall 0.404, flagged 134
class weights: AP 0.534 | at 0.5: precision 0.166, recall 0.850, flagged 1330
oversampling: AP 0.535 | at 0.5: precision 0.166, recall 0.850, flagged 1334
SMOTE: AP 0.512 | at 0.5: precision 0.168, recall 0.846, flagged 1309
undersampling: AP 0.526 | at 0.5: precision 0.170, recall 0.838, flagged 1285The untreated model is cautious: it flags 134 cases, 78% of them correctly, but finds only 40% of the positives. Each technique works on the training data differently:
- Class weights make each positive case count more in the loss.
- Random oversampling repeats positive cases.
- SMOTE creates synthetic positive cases along lines between neighbouring real ones (Chawla et al.: SMOTE).
- Random undersampling discards negative cases.
All four do nearly the same thing. They push predicted probabilities for positives upward, so at the 0.5 threshold recall jumps to about 85% while precision collapses to about 17%, with roughly ten times as many cases flagged. And their average precision falls, from 0.615 to between 0.51 and 0.54: on this data, none of them made the model better at ranking cases. Mostly they moved the decision boundary.
The simpler alternative: move the threshold
If the goal is higher recall, you can keep the untreated model and lower the threshold at which a case is flagged (scikit-learn: tuning the decision threshold). Comparing every model at the same recall of 85%:
def at_recall(proba, target=0.85):
precision, recall, thresholds = precision_recall_curve(y, proba)
i = np.where(recall[:-1] >= target)[0][-1]
return f"threshold {thresholds[i]:.3f}, precision {precision[i]:.3f}, flagged {(proba >= thresholds[i]).sum()}"
for name in ("no treatment", "class weights", "SMOTE"):
print(f"{name:>13} at recall 0.85:", at_recall(probas[name])) no treatment at recall 0.85: threshold 0.039, precision 0.219, flagged 1007
class weights at recall 0.85: threshold 0.500, precision 0.166, flagged 1328
SMOTE at recall 0.85: threshold 0.489, precision 0.162, flagged 1368To catch 85% of positives, the untreated model with a threshold of 0.039 flags 1,007 cases. The class-weighted and SMOTE models flag over 1,300 for the same catch rate: about 320 to 360 more false alarms for reviewers to clear. Threshold moving also leaves the model's probabilities meaningful, whereas rebalanced training inflates them, so a "50% risk" from a rebalanced model does not mean a one-in-two chance.
One caution: here the thresholds are read off the same out-of-fold predictions used for scoring, which is slightly optimistic for every model alike. In a real project, choose the threshold on validation data and confirm it on a held-out test set; scikit-learn's TunedThresholdClassifierCV automates that with cross-validation.
A better model beats a rebalanced one
print("gradient boosting:", end=" ")
boosted = evaluate(HistGradientBoostingClassifier(random_state=0))
print("gradient boosting at recall 0.85:", at_recall(boosted))gradient boosting: AP 0.728 | at 0.5: precision 0.807, recall 0.581, flagged 187
gradient boosting at recall 0.85: threshold 0.004, precision 0.301, flagged 735Gradient-boosted trees rank cases better, lifting average precision to 0.728, and at 85% recall they flag 735 cases instead of 1,007. No resampling technique came close to that improvement. When a rare class is hard to find, a better model and better features usually do more than rebalancing the data.
The mistake that fakes a great result
Resampling must happen inside cross-validation, on training folds only. The imblearn pipeline used above does exactly that. Here is what happens if SMOTE is applied to the whole dataset first:
X_wrong, y_wrong = SMOTE(random_state=0).fit_resample(X, y)
proba = cross_val_predict(make_pipeline(StandardScaler(), logistic()), X_wrong, y_wrong, cv=cv, method="predict_proba")[:, 1]
print("SMOTE before splitting, AP:", round(average_precision_score(y_wrong, proba), 3), "| rows:", len(y_wrong))SMOTE before splitting, AP: 0.944 | rows: 21846An average precision of 0.944, far above anything real. Two things went wrong. The test folds now contain synthetic points built from training points, so the model is tested on near-copies of what it learned. And the test data is balanced, half positive, so it no longer resembles reality, where 2.3% of cases are positive. The imbalanced-learn documentation warns against exactly this (imbalanced-learn: common pitfalls). Always evaluate on data with the real class balance.
When rebalancing is worth trying
The results above are for one dataset and are not a universal rule. Resampling and class weights can help when:
- The model or library has no usable probability output, so you cannot move a threshold.
- The training algorithm struggles to learn anything about the rare class at all, for example with very few positives per batch in neural network training.
- A fixed decision rule is imposed downstream and cannot be changed.
Treat them as options to test with the same cross-validation, not as a required step.
A practical order of work
- Measure with the right metrics: average precision and precision or recall at a chosen operating point, never accuracy alone.
- Use stratified splits so every fold contains positives.
- Try a stronger model and better features before anything else.
- Choose the threshold from the costs of false alarms and misses, on validation data.
- Test class weights or resampling inside a pipeline, and keep them only if they improve the metric that matters.
- Get more positive examples if you can; nothing helps as reliably as more real data about the rare class.
For extremely rare events with few labels, framing the problem as anomaly detection can be the better route.
Is SMOTE always better than random oversampling?
Should I balance the test set?
What is a good precision or recall for an imbalanced problem?
Do tree models need class weights?
The confusion matrix explained shows how moving the threshold reshapes the four kinds of outcome. Evaluation metrics and techniques for model evaluation are part of the model evaluation and optimisation module of the Data Science course.
Sources
- mammography (OpenML dataset 310) — www.openml.org
- SMOTE: Synthetic Minority Over-sampling Technique — Journal of Artificial Intelligence Research, 2002
- Common pitfalls and recommended practices (imbalanced-learn) — imbalanced-learn.org
- Tuning the decision threshold for class prediction (scikit-learn) — scikit-learn documentation
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Data Science?
Learn Data Science at GloryTecks in Hyderabad, with trainers who have industry experience, real-time projects and placement assistance.
Related articles
Data Scientist vs AI Engineer in 2027: Which Role Fits You?
Data scientists turn data into answers and predictive models; AI engineers build products with foundation models, such as retrieval systems and agents. This comparison covers the daily work, skills, tools, interviews and career paths of both roles in 2027, how certifications reflect the shift, and how to choose.
Data Science Roadmap 2027: Skills, Order and a 12-Month Plan
A data science roadmap for 2027 in four phases over twelve months: Python, pandas 3 and SQL; statistics and experiments; machine learning with honest evaluation; and LLMs, deployment and a portfolio. It covers what changed for data scientists in 2026, the tool versions to install and a month-by-month plan.
Machine Learning Skills That Still Matter in 2027, and What LLMs Changed
Language models changed how we build text applications, but most business prediction problems still run on tables, where classical machine learning is strong. These are the machine learning skills that still matter in 2027, a runnable example of why the right metric matters, and where LLMs now fit into ML work.