Real-Time Analytics: Architecture and Tools
How real-time analytics systems are built: the four layers of ingestion, stream processing, serving and consumption, the tools at each layer including Kafka, Spark, Flink and real-time OLAP databases such as Druid, Pinot and ClickHouse, latency budgets, common architectures, and the pitfalls that raise costs.
GloryTecks Team
Training & Placement Faculty
Last updated: 1 October 2026
"Real-time analytics" means answering questions about data that is seconds or minutes old: a live operations dashboard, a fraud score before a payment clears, a count of active users right now. Building it is less about one fast tool than about a chain of layers, each of which must keep up. This guide lays out the architecture, the tools at each layer, and the trade-offs that vendors' diagrams tend to omit.
First, how real-time?
The term covers very different requirements, and the architecture follows from which one you have:
| Need | Typical latency | Example |
|---|---|---|
| Near-real-time reporting | Minutes | Operations dashboard refreshed every five minutes |
| Real-time analytics | Seconds | Live monitoring, user-facing statistics |
| Real-time decisions | Milliseconds | Fraud check, recommendation during a request |
Minutes can be met with frequent micro-batches into a warehouse. Seconds need a streaming pipeline and a store built for fresh data. Milliseconds are an application concern, served from precomputed features and key-value lookups. Before designing anything, establish which row you are in; batch vs streaming measures what each step in freshness actually buys.
The four layers
Microsoft's reference architecture names the components: message ingestion, stream processing, an analytical data store, and analysis and reporting (Microsoft Learn: big data architectures).
1. Ingestion: a durable event log
Events come from applications, devices and databases, the last through change data capture. They land in a log that buffers them, so producers and consumers are decoupled and a slow consumer does not lose data. Kafka is the usual choice: it publishes, stores and processes streams of events and retains them so many consumers can read independently (Kafka docs: introduction). Cloud equivalents include Amazon Kinesis, Azure Event Hubs and Google Pub/Sub. Apache Kafka explained covers the log itself.
2. Stream processing: shape the events
Raw events are rarely what a dashboard needs. A stream processor parses, filters, joins with reference data and aggregates over time windows. Spark Structured Streaming and Apache Flink are the common engines; Kafka Streams is a lighter option embedded in an application. This layer is where correctness is decided: event time versus arrival time, watermarks for late events, and exactly how duplicates are absorbed. Building a streaming pipeline with Kafka and Spark shows a windowed aggregation and why its newest window stays open.
3. Serving: a store that is fast on fresh data
A dashboard needs to query the results in well under a second while new data keeps arriving. Three families serve this:
- Real-time OLAP databases. Apache Druid describes itself as a real-time analytics database for fast slice-and-dice queries, with real-time ingestion, time-based partitioning and summarisation at ingest (Druid docs). Apache Pinot is a real-time distributed OLAP datastore built at LinkedIn for user-facing analytics, ingesting directly from Kafka (Pinot docs). ClickHouse is a column-oriented SQL database for analytical queries that return in under a second (ClickHouse docs). Stores of this kind can ingest from the event log directly, which sometimes removes the need for a separate processing layer.
- Lakehouse and warehouse tables fed by streaming writes, for latencies of a minute or more, reusing the platform a team already runs.
- Key-value stores, for single-record lookups by an application.
4. Consumption: dashboards, alerts, applications
The last layer is where latency is finally felt. A dashboard that refreshes every five minutes wastes a pipeline that delivers in two seconds. Alerts should come from the stream or the serving store with clear thresholds; applications read through an API with its own caching.
Two common shapes
- Log, processor, OLAP store: Kafka, then Spark or Flink for enrichment, then Druid, Pinot or ClickHouse. Flexible and fast, with three systems to run.
- Log straight into an OLAP store: events are ingested directly and aggregated at query time or by the store's own roll-ups. Fewer parts, less room for complex joins.
Both usually keep a second path: the same events written to cheap storage for history and reprocessing, the idea behind the designs in Lambda vs Kappa.
A latency budget
End-to-end delay is the sum of every hop: time to emit the event, time in the log, processing and its trigger interval, any window and watermark delay, ingestion into the store, query time, and dashboard refresh. Write down a target for each. The largest term is very often not the technology but a setting, a one-minute trigger, a ten-minute watermark or a slow refresh, and fixing that costs nothing.
Pitfalls
- Real-time by default. Each step towards lower latency adds always-on compute and operational load. Many "real-time" requests are satisfied by fifteen-minute data.
- Ignoring late and duplicate events, so live numbers disagree with tomorrow's batch report. Microsoft's guidance lists checkpointing, idempotent transformations, watermarks and dead-letter queues as the baseline.
- Unbounded cardinality: grouping by user ID or URL in a live store can exhaust memory. Decide dimensions deliberately and use approximate counts where exactness is not needed.
- No reconciliation. Compare streaming totals with a batch recomputation daily, so drift is found by you and not by a customer.
- No replay plan. Bugs happen; keep raw events long enough to rebuild.
What is real-time analytics?
Which database is used for real-time analytics?
Do I need Kafka for real-time analytics?
Is real-time analytics expensive?
Stream processing concepts, Apache Kafka basics, real-time data pipelines and a real-time streaming pipeline project are part of the Data Engineering course.
Sources
- Big data architectures (Azure Architecture Center, Microsoft Learn) — Microsoft Learn
- Extract, transform, load (ETL) (Azure Architecture Center, Microsoft Learn) — Microsoft Learn
- Introduction (Apache Kafka documentation) — Apache Kafka
- Introduction to Apache Druid (Apache Druid documentation) — Apache Druid
- Apache Pinot documentation — Apache Pinot
- What is ClickHouse? (ClickHouse documentation) — ClickHouse
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Data Engineering?
Learn Data Engineering at GloryTecks in Hyderabad, with trainers who have industry experience, real-time projects and placement assistance.
Related articles
Data Warehouse Explained: Concepts and Architecture
A clear explanation of data warehouses: why analytical queries need a separate store from transactional systems, the architecture from sources through staging, modelled layers and a semantic model, facts, dimensions, grain and history, modelling approaches, modern platforms and common pitfalls.
Medallion Architecture: Bronze, Silver and Gold Layers
The medallion architecture explained by building it: bronze as raw data exactly as it arrived, silver as validated and enriched rows, a quarantine table that keeps every rejected row with its reason, gold shaped for one dashboard, a reconciliation that bronze equals silver plus quarantine, and the mistakes teams make.
Apache Iceberg v3: What Data Engineers Should Know for 2027
Apache Iceberg's format version 3 adds deletion vectors, row lineage, default column values and new types such as variant, geospatial and nanosecond timestamps, and in 2026 Snowflake and AWS made it generally available. What v3 changes, why it matters, which engines support it and how to upgrade a table safely.