Data Engineering Projects for Your Portfolio
Five data engineering projects for a portfolio, each on a real public dataset: a tested batch pipeline and warehouse on NYC taxi data, an orchestrated ingestion of live earthquake feeds, change capture with history on a sample store database, a streaming pipeline on Wikipedia edits, and a Spark job on GitHub events.
GloryTecks Team
Training & Placement Faculty
Last updated: 1 October 2026
A portfolio project has one job: to let a stranger see that you can build a pipeline that works and explain why you built it that way. Tutorials copied line for line do not do that, and neither do toy datasets with no faults in them. The five projects below use real public data, with real problems, and each demonstrates a different part of the role. Two done properly are worth more than five half-finished.
What makes a project credible
- Real data, large or messy enough to need engineering.
- Reliability shown, not claimed: a rerun that does not duplicate, tests that fail when data is bad, a backfill.
- A clear README: the question, the design, a diagram, how to run it, and what you would change at ten times the scale.
- Honest limits. Say what is not handled.
Project 1: a tested batch pipeline and warehouse
Data: New York yellow taxi trip records, published monthly as Parquet files with a zone lookup table (NYC TLC: trip record data). A month is about three million rows, and the TLC itself warns that it makes no representations about accuracy, so the data has genuine faults: trips dated outside their month, negative fares, trips that end before they start.
Build: ingest several months; keep a raw layer; apply named rejection rules and count what each rejects; model a star schema with a trip fact and date, zone and payment dimensions; test for orphan keys and reconcile row counts; make the monthly load idempotent.
It proves: ingestion, data quality, dimensional modelling and safe reruns, the core of the job. Building your first data pipeline and how to build a data warehouse show each half on this dataset.
Stretch: load 2022 to 2025 and handle the schema changes between years, including a renamed column and a fee column that only exists from 2025.
Project 2: an orchestrated API ingestion
Data: the U.S. Geological Survey's earthquake feeds, GeoJSON summaries for the past hour, day, week and month, intended for programs and updated every minute (USGS: GeoJSON feed).
Build: an Airflow DAG that fetches the feed on a schedule, lands the raw JSON by date, flattens it into a table, and merges on the event ID. Each event carries an updated timestamp and a status that moves from automatic to reviewed, so the same event returns with changed values; appending would duplicate it, and the pipeline must upsert. Add a freshness check and an alert when the feed returns nothing.
It proves: API extraction, scheduling, semi-structured data, upserts and monitoring. The Airflow tutorial provides the DAG pattern.
Stretch: a small dashboard of events by region and magnitude, refreshed by the pipeline.
Project 3: change capture and history
Data: the Chinook sample database, a digital media store with customers, invoices, tracks and employees (Chinook on GitHub). It is small, which is the point: this project is about correctness over time, not volume.
Build: treat Chinook as a live operational database. Write a script that makes changes to it, new invoices, a customer moving country, a deleted record. Capture those changes, replicate them to an analytical copy, and model the customer dimension as Type 2 so history is kept. Then show a report whose numbers differ depending on whether history is preserved.
It proves: change data capture, deletes, idempotent replay and slowly changing dimensions, topics that separate candidates in interviews. CDC explained builds the capture and apply steps.
Stretch: move the source to PostgreSQL and use log-based capture.
Project 4: a streaming pipeline
Data: Wikimedia's EventStreams, a public service that exposes continuous streams of structured events over HTTP, including every edit to Wikipedia as it happens (Wikimedia: EventStreams). It is a real, endless, uneven stream, and it is meant for small-scale external tools, so be a polite consumer.
Build: a producer that reads the stream and writes events to a Kafka topic; a stream processor that counts edits per wiki per minute using event time and a watermark; a sink table; a way to replay from the topic when your logic changes.
It proves: event logs, keys and partitions, windows, late data and at-least-once processing. Building a streaming pipeline with Kafka and Spark is the template.
Stretch: detect bursts of edits to a single page, and write the alerts to a second topic.
Project 5: Spark at scale
Data: GH Archive, which records the public GitHub timeline as hourly JSON archives, available since 2011 (GH Archive). When we checked, the 24 files for 15 September 2026 came to about 1 GB compressed, and the midday file expanded to 4.6 times its size as nested JSON, with 55,822 events in that hour. A week at that rate is enough that a single-process script strains and a distributed engine earns its keep.
Build: download a week of hourly files; parse the nested JSON with an explicit schema in PySpark; write partitioned Parquet or a Delta table; compute repository activity metrics; then read the query plan and improve one thing, a partitioning choice, a broadcast join, a pruned column, and document the before and after.
It proves: semi-structured data at volume, file and table formats, partitioning, and performance reasoning from evidence. Apache Spark explained and Delta Lake explained cover the foundations.
Stretch: run it on a cloud platform and report the cost.
Presenting the work
- One repository per project, with a README a busy reviewer can skim in two minutes.
- A diagram of sources, layers and schedule.
- Evidence: test output, a screenshot of the orchestrator's run history, row counts before and after rules.
- A "decisions" section: grain, incremental strategy, partitioning, and what you rejected.
- A "limits" section: what breaks at scale and what you would do next.
Be ready to be asked, for any line of it, "why?". That conversation is the real purpose of the portfolio.
How many projects do I need in a data engineering portfolio?
Can I use Kaggle datasets?
Do portfolio projects need to run in the cloud?
What do reviewers look for first?
An end-to-end ETL pipeline, a cloud data warehouse, a real-time streaming pipeline and a data lake implementation are the projects built in the Data Engineering course.
Sources
- TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission
- GeoJSON summary feed (U.S. Geological Survey) — U.S. Geological Survey
- Chinook sample database (GitHub) — GitHub
- EventStreams HTTP service (Wikimedia) — Wikimedia
- GH Archive — GH Archive
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Data Engineering?
Learn Data Engineering at GloryTecks in Hyderabad, with trainers who have industry experience, real-time projects and placement assistance.
Related articles
Data Engineering Roadmap 2026: A Complete Guide
A stage-by-stage data engineering roadmap for 2026: what the role involves according to current certification guides, then seven stages, SQL, Python, pipelines and modelling, orchestration and quality, Spark and big data, streaming, and cloud platforms, each with a checkpoint, and detours to avoid.
Data Warehouse Explained: Concepts and Architecture
A clear explanation of data warehouses: why analytical queries need a separate store from transactional systems, the architecture from sources through staging, modelled layers and a semantic model, facts, dimensions, grain and history, modelling approaches, modern platforms and common pitfalls.
Data Lake vs Data Warehouse vs Lakehouse
A clear comparison of data lakes, data warehouses and lakehouses: what each stores and when structure is applied, strengths and failure modes such as the data swamp, what a lakehouse adds through table formats and governance, a side-by-side table, and practical guidance on choosing or combining them.