Anomaly Detection: Techniques and Real-World Use Cases
How anomaly detection works in practice: rules and thresholds, z-scores and the masking problem that lets outliers hide themselves, robust median-based scores, Isolation Forest, anomalies in time series, and how to evaluate a detector when almost nothing is labelled, with fraud, sensor and data-quality use cases.
GloryTecks Team
Training & Placement Faculty
Last updated: 25 September 2026
An anomaly is a record that does not fit the pattern of the rest: an order of 15 rupees in a shop where orders cluster around 1,200 rupees, a sensor reading that jumps tenfold, a day with no sales at all. Sometimes it is the most important record in the dataset (fraud, a failing machine, a broken pipeline); sometimes it is a data-entry error. Either way you want to find it. The difficulty is that "normal" is rarely defined, and labelled examples of anomalies are almost never available.
Start with rules you can explain
Before any statistics, write down what is impossible or clearly suspicious: negative quantities, a delivery date before the order date, more than ten orders from one card in an hour, a temperature outside the sensor's physical range. Rules catch known problems exactly, are easy to explain to the people who act on the alerts, and cost nothing to run. Statistical and machine-learning methods are for the unknown problems the rules do not anticipate.
Z-scores, and how outliers hide themselves
The classic statistical test flags values more than three standard deviations from the mean. It has a weakness that the simulation below exposes: anomalies distort the mean and standard deviation they are measured against. The data is 990 ordinary transaction amounts around 1,200 rupees plus ten injected anomalies, eight very large and two tiny:
import numpy as np
from sklearn.ensemble import IsolationForest
rng = np.random.default_rng(42)
normal = rng.normal(1200, 150, 990)
anomalies = np.array([9000, 9500, 12000, 11000, 8800, 10500, 15, 20, 9900, 13000])
amounts = np.concatenate([normal, anomalies])
is_anomaly = np.r_[np.zeros(990, bool), np.ones(10, bool)]
z = (amounts - amounts.mean()) / amounts.std()
flag_z = np.abs(z) > 3
median = np.median(amounts)
mad = np.median(np.abs(amounts - median))
robust_z = 0.6745 * (amounts - median) / mad
flag_mad = np.abs(robust_z) > 3.5
iso = IsolationForest(contamination=0.01, random_state=0).fit(amounts.reshape(-1, 1))
flag_iso = iso.predict(amounts.reshape(-1, 1)) == -1
for name, flags in [("z-score > 3", flag_z), ("robust z (MAD) > 3.5", flag_mad), ("Isolation Forest", flag_iso)]:
caught = (flags & is_anomaly).sum()
false_alarms = (flags & ~is_anomaly).sum()
print(f"{name:<22} caught {caught}/10 false alarms {false_alarms}")
print(f"std with anomalies {amounts.std():.1f}, without {normal.std():.1f}")z-score > 3 caught 8/10 false alarms 0
robust z (MAD) > 3.5 caught 10/10 false alarms 1
Isolation Forest caught 10/10 false alarms 0
std with anomalies 849.6, without 148.3The last line explains the first. The eight huge values inflated the standard deviation from about 148 to about 850, so the 15- and 20-rupee transactions, which are wildly unusual, no longer looked extreme by that yardstick. This is called masking: outliers hide other outliers.
Robust scores: use the median instead
The median and the median absolute deviation (MAD) barely move when a few extreme values are added, so a z-score built on them keeps its yardstick honest. The modified z-score multiplies by 0.6745 so it is comparable with a standard z-score for normal data, and a cut-off of 3.5 is the value NIST's statistics handbook cites as recommended (NIST: detection of outliers). It caught all ten, at the cost of one false alarm: a legitimately unusual but genuine transaction. For a single numeric column, this is the method to try first.
Isolation Forest: many features at once
Real anomalies are often unusual only in combination: a normal amount, at a normal time, but from a new device in a different country. Isolation Forest handles many features together. It builds random trees that split the data at random points; anomalies, being few and different, get isolated in fewer splits than normal points (scikit-learn: IsolationForest). It needs no labels and scales to large datasets.
Its perfect score above comes with a caveat: contamination=0.01 told it to flag about 1% of the data, and exactly 1% was anomalous by construction. In real data the true rate is unknown, so either tune the threshold on the scores against a sample you have reviewed, or rank records by anomaly score and review the top of the list. scikit-learn documents several alternatives, including Local Outlier Factor for anomalies that are unusual relative to their neighbourhood (scikit-learn: outlier detection).
Anomalies in time series
Time adds context: 500 orders at 3 a.m. might be an anomaly, while 500 at noon is normal. Compare each point with what was expected for that moment, not with the overall average. A practical approach is to model the expected value, using a rolling median of the same hour on previous days or a forecasting model, and flag points whose residual, the gap between actual and expected, is extreme by a robust score. Time series forecasting covers building the expected-value model.
Evaluating a detector without labels
Accuracy is meaningless here: a detector that flags nothing is 99% accurate when 1% of records are anomalies. Useful measures are:
- Precision of the alerts: of the records flagged, how many were real problems once someone reviewed them. Alert fatigue kills detectors faster than missed anomalies do.
- Recall on known cases: replay past incidents, such as confirmed fraud or known outages, and check how many the detector would have caught.
- Precision at k: with a daily review budget of, say, 50 records, what share of the top 50 by anomaly score were real.
Evaluation metrics: accuracy, precision, recall and F1 explains these measures in detail, and the imbalance problem connects to handling imbalanced datasets when some labels do exist.
Where anomaly detection is used
- Payments and fraud: unusual amounts, locations, devices or velocities for an account.
- Industrial sensors: vibrations or temperatures drifting away from a machine's normal profile before a failure.
- IT operations: sudden changes in error rates, latency or traffic.
- Data quality: a daily file with half the usual rows, a column suddenly full of nulls, a price column that jumps a hundredfold, which is often the cheapest and most valuable use in data teams.
Should I remove outliers before training a model?
What is the difference between outlier and novelty detection?
How often should an anomaly detector be retrained?
Can deep learning detect anomalies?
Anomaly detection is one of the applied specialisations built on the machine learning and statistics taught in the Data Science course, alongside its exploratory data analysis module, which includes outlier detection.
Sources
- Novelty and Outlier Detection (scikit-learn) — scikit-learn documentation
- sklearn.ensemble.IsolationForest — scikit-learn documentation
- NIST: Detection of Outliers — NIST
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Data Science?
Learn Data Science at GloryTecks in Hyderabad, with trainers who have industry experience, real-time projects and placement assistance.
Related articles
Data Scientist vs AI Engineer in 2027: Which Role Fits You?
Data scientists turn data into answers and predictive models; AI engineers build products with foundation models, such as retrieval systems and agents. This comparison covers the daily work, skills, tools, interviews and career paths of both roles in 2027, how certifications reflect the shift, and how to choose.
Data Science Roadmap 2027: Skills, Order and a 12-Month Plan
A data science roadmap for 2027 in four phases over twelve months: Python, pandas 3 and SQL; statistics and experiments; machine learning with honest evaluation; and LLMs, deployment and a portfolio. It covers what changed for data scientists in 2026, the tool versions to install and a month-by-month plan.
Machine Learning Skills That Still Matter in 2027, and What LLMs Changed
Language models changed how we build text applications, but most business prediction problems still run on tables, where classical machine learning is strong. These are the machine learning skills that still matter in 2027, a runnable example of why the right metric matters, and where LLMs now fit into ML work.