Apache Spark Explained for Beginners
A beginner's guide to Apache Spark: what it is for, how the driver, executors and cluster manager fit together, jobs, stages and tasks, lazy transformations and actions demonstrated on three million New York taxi trips, partitions, installing PySpark, and the mistakes beginners make.
GloryTecks Team
Training & Placement Faculty
Last updated: 26 September 2026
Apache Spark is, in its own documentation's words, a unified analytics engine for large-scale data processing (Spark docs: overview). In practice it lets you write one program, in Python, SQL, Scala or R, and run it across many machines, each working on part of the data. This guide explains how that works, shows Spark's most important behaviour on three million real taxi trips, and is honest about when you do not need Spark at all.
When Spark is the right tool
Spark earns its complexity when data is too big for one machine's memory, when a job would take hours on one machine, or when a team already runs its pipelines on a Spark platform such as Databricks, Amazon EMR or Google Dataproc. For a file of a few hundred megabytes on a laptop, pandas, Polars or DuckDB is usually simpler and faster; PySpark vs pandas compares them. The example below is deliberately small so you can run it anywhere; the concepts are identical at a thousand times the size.
How Spark runs a program
Spark's cluster overview describes a small set of moving parts (Spark docs: cluster overview):
- Driver: the process running your program's main function. It builds the plan and schedules work.
- Executors: processes on worker machines that run tasks and hold data in memory or on disk. Each application gets its own.
- Cluster manager: allocates machines. Spark supports its own standalone manager, Hadoop YARN and Kubernetes.
- Job, stage, task: an action, such as writing or collecting results, triggers a job; the job is split into stages; each stage runs as tasks, one per partition of data.
In local mode, used below, the driver and executor share one process on your machine, with worker threads standing in for executors. It is how most people learn Spark and test code before running it on a cluster.
Lazy evaluation, seen on real data
The single most important idea in Spark is laziness. Transformations, such as filter, group or join, are not computed right away; Spark remembers them and computes only when an action needs a result (Spark docs: programming guide). You can watch this happen by counting the jobs Spark has run. The data is January 2024 New York yellow taxi trips, published by the city's Taxi and Limousine Commission as Parquet files with a zone lookup table (NYC TLC: trip record data):
import urllib.request
from pathlib import Path
from pyspark.sql import SparkSession, functions as F
base = "https://d37ci6vzurychx.cloudfront.net"
for name, url in {"yellow_tripdata_2024-01.parquet": f"{base}/trip-data/yellow_tripdata_2024-01.parquet",
"taxi_zone_lookup.csv": f"{base}/misc/taxi_zone_lookup.csv"}.items():
if not Path(name).exists():
urllib.request.urlretrieve(url, name)
spark = SparkSession.builder.master("local[4]").appName("taxi").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
jobs = lambda: len(spark.sparkContext.statusTracker().getJobIdsForGroup())
trips = spark.read.parquet("yellow_tripdata_2024-01.parquet")
print("jobs after reading the Parquet file:", jobs())
zones = spark.read.csv("taxi_zone_lookup.csv", header=True, inferSchema=True)
print("jobs after reading the CSV with inferSchema:", jobs())
busiest = (trips.where(F.col("fare_amount") >= 0)
.groupBy("PULocationID")
.agg(F.count("*").alias("trips"), F.round(F.avg("fare_amount"), 2).alias("avg_fare"))
.join(zones, F.col("PULocationID") == F.col("LocationID"))
.select("Zone", "Borough", "trips", "avg_fare")
.orderBy(F.desc("trips")))
print("jobs after transformations:", jobs())
busiest.show(5, truncate=False)
print("jobs after show():", jobs())
print("input partitions:", trips.rdd.getNumPartitions())
spark.stop()jobs after reading the Parquet file: 1
jobs after reading the CSV with inferSchema: 3
jobs after transformations: 3
+---------------------+---------+------+--------+
|Zone |Borough |trips |avg_fare|
+---------------------+---------+------+--------+
|Midtown Center |Manhattan|141749|15.58 |
|Upper East Side South|Manhattan|141270|12.42 |
|JFK Airport |Queens |141196|62.46 |
|Upper East Side North|Manhattan|135339|12.92 |
|Midtown East |Manhattan|105469|15.14 |
+---------------------+---------+------+--------+
only showing top 5 rows
jobs after show(): 6
input partitions: 4Read the job counts in order:
- Reading Parquet ran one small job, to read the schema stored in the file. Parquet files carry their own schema, so Spark did not scan the data.
- Reading the CSV ran two more. A CSV has no stored types, and inferSchema requires one extra pass over the data (Spark docs: CSV). On a 12 KB lookup file that is trivial; on a 500 GB CSV it is an expensive full scan before any real work, which is why production code declares schemas.
- Filter, group, join, select and sort ran nothing. The count stayed at three: Spark only built a plan.
- show() triggered the work, three jobs covering the scan, aggregation, join and sort, because it needed rows to display.
Laziness is what lets Spark optimise: it sees the whole chain before running any of it, so it can push the fare filter into the Parquet scan and read only the columns it needs. The result is also a useful sanity check on the data: JFK Airport has almost as many pickups as Midtown, at four times the average fare.
Transformations and actions
| Transformations: build the plan | Actions: run it |
|---|---|
| select, where or filter, withColumn | show, count, collect |
| groupBy with agg, join, union | write, save |
| orderBy, distinct, limit | take, first, toPandas |
A practical consequence: an error in a transformation often appears only when an action runs, several lines later. And each action recomputes the plan from the source unless you cache an intermediate result, which the programming guide describes with persist and cache.
Partitions: the unit of parallelism
Spark splits data into partitions and runs one task per partition. The taxi file became four input partitions, one per thread of local[4]. On a cluster, the number and size of partitions decide how evenly work spreads across executors, and poor partitioning is behind many slow jobs; partitioning and bucketing in Spark covers how to control it.
Getting started
- Install:
pip install pysparkgives you local mode. Spark 3.5 runs on Java 8, 11 or 17; the Spark 4 releases require Java 17 or later, so check your Java version first. - Use DataFrames and Spark SQL, not the older RDD API, for almost all work; they are optimised automatically.
- Learn the API with the PySpark tutorial, then managed platforms with the Databricks tutorial.
Mistakes beginners make
- collect() on large data, which pulls everything to the driver and can crash it.
- inferSchema on big files, an extra full scan; declare the schema instead.
- Python UDFs for things built-in functions do, which move data between the JVM and Python and are usually much slower.
- Using Spark for small data, paying cluster start-up and coordination costs for a job pandas finishes in seconds.
When jobs get slow, Spark performance tuning covers shuffles, joins and caching, and Spark vs Hadoop explains where Spark came from.
Is Spark a database?
Do I need a cluster to learn Spark?
What is the difference between a transformation and an action?
Should I learn PySpark or Scala Spark?
Spark architecture, PySpark basics, DataFrames and RDDs, Spark SQL and performance optimization are part of the Apache Spark module of the Data Engineering course.
Sources
- Spark 3.5.7 overview (Apache Spark documentation) — Apache Spark
- Cluster mode overview (Apache Spark documentation) — Apache Spark
- RDD programming guide (Apache Spark documentation) — Apache Spark
- CSV files (Spark SQL documentation) — Apache Spark
- TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Data Engineering?
Learn Data Engineering at GloryTecks in Hyderabad, with trainers who have industry experience, real-time projects and placement assistance.
PySpark Tutorial: DataFrames, Transformations and Actions
NextMedallion Architecture: Bronze, Silver and Gold Layers
Related articles
Data Engineering Roadmap 2027: Lakehouse, Streaming and AI-Ready Data
A data engineering roadmap for 2027 in seven stages: SQL and modelling, Python, one cloud, Spark and the lakehouse, Airflow 3, streaming and change data capture, and data quality, governance and AI-ready data. Includes the 2026 platform changes, a nine-month plan and three portfolio projects.
Spark vs Hadoop: What Changed and Why
What changed when Spark overtook Hadoop MapReduce: the three parts of Hadoop, the MapReduce model emulated step by step on three million taxi trips with its intermediate data written to disk, why in-memory processing and DAGs mattered, what Spark still borrows from Hadoop, and what that means today.
Apache Iceberg v3: What Data Engineers Should Know for 2027
Apache Iceberg's format version 3 adds deletion vectors, row lineage, default column values and new types such as variant, geospatial and nanosecond timestamps, and in 2026 Snowflake and AWS made it generally available. What v3 changes, why it matters, which engines support it and how to upgrade a table safely.