Bias-Variance Tradeoff Explained with Real Examples
Bias and variance measured rather than described: a simulation that fits models of increasing complexity to 200 resampled training sets and splits their error into bias, variance and noise, then what high bias and high variance look like in real projects, and the levers that fix each.
GloryTecks Team
Training & Placement Faculty
Last updated: 25 September 2026
Every model's error on new data comes from three sources. Bias is error from wrong assumptions: a straight line fitted to a curve is wrong no matter how much data you give it. Variance is error from sensitivity to the particular training sample: a model that bends to fit every point changes shape every time the sample changes. Noise is randomness in the data itself, which no model can remove. Making a model more flexible lowers bias and raises variance; the tradeoff is finding the flexibility where their sum is smallest.
That description is familiar. It is more convincing measured, so here is an experiment that measures it.
The experiment
The true relationship is a sine curve, and each training set is 40 noisy observations of it. For each model complexity, polynomial degree 1, 3, 5 and 9, the code draws 200 independent training sets, fits a model to each, and records its predictions at 50 fixed test points (scikit-learn: PolynomialFeatures). The average prediction across the 200 models shows systematic error, bias; the spread of predictions shows variance.
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
rng = np.random.default_rng(0)
def true_f(x):
return np.sin(2 * np.pi * x)
x_test = np.linspace(0.1, 0.9, 50)
noise = 0.3
for degree in (1, 3, 5, 9):
preds = []
for _ in range(200): # 200 independent training sets
x = rng.uniform(0, 1, 40)
y = true_f(x) + rng.normal(0, noise, 40)
model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(x[:, None], y)
preds.append(model.predict(x_test[:, None]))
preds = np.array(preds)
bias2 = np.mean((preds.mean(axis=0) - true_f(x_test)) ** 2)
variance = np.mean(preds.var(axis=0))
print(f"degree {degree}: bias^2 {bias2:.3f} variance {variance:.3f} total error {bias2 + variance + noise**2:.3f}")degree 1: bias^2 0.151 variance 0.014 total error 0.255
degree 3: bias^2 0.003 variance 0.007 total error 0.101
degree 5: bias^2 0.000 variance 0.012 total error 0.103
degree 9: bias^2 0.001 variance 0.142 total error 0.233The total is the expected squared error on new data: bias squared plus variance plus the noise variance of 0.09, which no model can go below.
Reading the results
- Degree 1 (a straight line) has almost all its error in bias. Its 200 fitted lines agree closely with each other (low variance) and are all wrong in the same way, because a line cannot follow a sine wave. More data would not help.
- Degree 3 removes nearly all the bias while keeping variance low. Its total error, 0.101, is barely above the noise floor of 0.09: this is the sweet spot for this problem.
- Degree 5 is about as good: its bias is essentially zero, and its slight extra variance is the price of flexibility it did not need.
- Degree 9 has almost no bias but ten times the variance of degree 3. Each model bends to fit its own 40 noisy points, so the 200 models disagree wildly between observations, and the total error more than doubles.
That is the tradeoff in numbers: as complexity rises, bias falls and variance rises, and total error traces a U. scikit-learn's own underfitting and overfitting example shows the same effect visually (scikit-learn: underfitting vs overfitting).
Recognising each in a real project
You never have 200 training sets, and you never know the true function. You do have training and validation scores, and their pattern tells you which problem dominates (scikit-learn: validation and learning curves):
| Symptom | Likely problem |
|---|---|
| Training error high, validation error similar and high | High bias (underfitting) |
| Training error low, validation error much higher | High variance (overfitting) |
| Both low and close together | A good fit for the data you have |
A learning curve, error plotted against training-set size, adds one more clue. With high bias, both curves flatten out early at a high error, and more data does not help. With high variance, the gap between them narrows as data grows, so more data is a real fix.
A diagnosis, step by step
Suppose a churn model built with logistic regression on five customer attributes scores about the same on training and validation data, and both scores are disappointing. The two numbers being close rules out overfitting, so variance is not the problem, and collecting more of the same data will not help. The model is too simple for the pattern, so the right moves are richer features, such as usage trends over time instead of a single snapshot, or a more flexible model. Now suppose a gradient-boosted model on 200 features scores almost perfectly on training data and much worse on validation. That gap is variance: reduce the number of features, constrain the trees, add regularisation or add data, and watch whether the gap closes.
Fixing high bias
- Use a more flexible model: add features or interactions, use a non-linear model such as a tree ensemble, or reduce regularisation.
- Engineer better features. A model cannot use information it is never given; see feature engineering.
- Train longer, for iterative models that stopped too early.
Fixing high variance
- Collect more training data, if the learning curve says it will help.
- Increase regularisation: a larger penalty in ridge or lasso regression, fewer leaves or a shallower depth in trees, dropout in neural networks.
- Reduce the number of features, removing noisy or redundant ones.
- Average many models. Bagging and random forests reduce variance by averaging trees trained on different samples; ensemble learning explains why that works.
Choose the fix after diagnosing, not before. Adding data to a high-bias model and making a high-variance model more complex are the two most common ways to spend a week without improving anything.
A note on modern models
Very large models, such as deep networks with more parameters than training examples, can generalise well despite enormous capacity, which the simple U-curve does not predict. Regularisation, training procedures and data scale all play a part. For the tabular problems most data scientists work on, the classical picture above remains a reliable guide to diagnosis.
Is bias in the bias-variance tradeoff the same as unfair bias?
How do I measure bias and variance on real data?
Does regularisation always reduce variance?
The practical fixes for each side are collected in overfitting and underfitting: causes and fixes. Model evaluation, including overfitting and underfitting, is part of the model evaluation and optimisation module of the Data Science course.
Sources
- Underfitting vs. Overfitting (scikit-learn example) — scikit-learn documentation
- Validation curves and learning curves (scikit-learn) — scikit-learn documentation
- PolynomialFeatures — scikit-learn documentation
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Data Science?
Learn Data Science at GloryTecks in Hyderabad, with trainers who have industry experience, real-time projects and placement assistance.
Related articles
Data Scientist vs AI Engineer in 2027: Which Role Fits You?
Data scientists turn data into answers and predictive models; AI engineers build products with foundation models, such as retrieval systems and agents. This comparison covers the daily work, skills, tools, interviews and career paths of both roles in 2027, how certifications reflect the shift, and how to choose.
Data Science Roadmap 2027: Skills, Order and a 12-Month Plan
A data science roadmap for 2027 in four phases over twelve months: Python, pandas 3 and SQL; statistics and experiments; machine learning with honest evaluation; and LLMs, deployment and a portfolio. It covers what changed for data scientists in 2026, the tool versions to install and a month-by-month plan.
Machine Learning Skills That Still Matter in 2027, and What LLMs Changed
Language models changed how we build text applications, but most business prediction problems still run on tables, where classical machine learning is strong. These are the machine learning skills that still matter in 2027, a runnable example of why the right metric matters, and where LLMs now fit into ML work.