Apache Spark: MapReduce Evolved

Concept. Spark generalizes MapReduce by representing a dataset as a Resilient Distributed Dataset (RDD): a partitioned, lazily-evaluated, lineage-tracked collection. You chain many transformations into a single in-memory pipeline that recomputes lost partitions from lineage instead of re-reading from disk.

Intuition. When a data scientist explores last year's Listens with filter(genre == 'pop').groupBy(user_id).count().orderBy(desc('count')).limit(100), Spark builds a single in-memory pipeline across thousands of cores, holds intermediate results in RAM (so the next exploratory query takes seconds, not minutes), and if a machine dies mid-job, Spark recomputes only that machine's partition from the lineage.

RDDs: The In-Memory Pipeline

Spark's core abstraction is the RDD, a Resilient Distributed Dataset: a collection split into partitions across the cluster, held in memory, and built by chaining transformations like filter and map. Spark does not run each transformation eagerly. It records the chain as a lineage graph and evaluates it lazily, in one pass, only when an action finally asks for a result. So intermediate data stays in RAM between stages instead of round-tripping to disk, which is the whole reason the same three-stage pipeline (filter to pop, count by user, sort the top) runs about 100× faster than MapReduce.

Two panels: MapReduce runs a three-stage pipeline as three chained jobs, each writing its output to pink HDFS before the next job reads it, in about 3 minutes; Spark chains the same three stages into one lazy in-memory pipeline that keeps RDDs in pink RAM with zero disk writes in about 2 seconds

Figure 1. The same three-stage analytics pipeline (filter to pop, count by user, sort the top), two engines. MapReduce runs it as three chained jobs and materializes each job's output to HDFS before the next reads it (about 3 minutes); Spark chains the same three stages but keeps intermediate RDDs in memory and evaluates lazily, building a plan it runs once when an action triggers it (about 2 seconds). The ~100× gap comes from data staying in RAM between stages, operator fusion from lazy evaluation, and cheap lineage-based recovery instead of replication.

Resilience: Lineage, Not Replicas

The R in RDD is resilient. RAM is volatile, so if a node crashes mid-job Spark could re-read from disk (slow) or keep three copies of every intermediate (expensive). It does neither: it keeps the recipe, the lineage, not copies of the data.

Four RDD stages with grey partitions, all narrow transformations; the worker holding RDD3.P1 dies so that partition is pink, and Spark replays filter then map then map on just the still-alive upstream RDD0.P1 along the pink path, with no replicas stored

Figure 2. Spark recovers lost data by replaying lineage instead of keeping replicas. Each RDD records the transformation that produced it (filter, map, map), so Spark stores the operator graph, not the intermediate data. These are all narrow transformations, so each output partition depends on exactly one upstream partition: when a worker dies and takes RDD3.P1 with it, Spark re-runs filter → map → map on just the still-alive RDD0.P1. Storage drops from 3× to 1×, and the cost of a failure rises from reading a replica to recomputing one lost partition. A wide transformation like groupBy would instead re-read every upstream partition through a shuffle.

This works only because Spark's transformations are deterministic: same input, same operator, same output, so replaying the lineage reproduces the lost partition exactly.

Spark SQL: DataFrames + Catalyst

Most teams do not write low-level RDD code anymore. They write SQL or DataFrame transformations, and Spark's Catalyst optimizer compiles them into the underlying RDD operations.

The Spotify genre-by-week join query on top, Catalyst's grey execution plan lower-left, and a pink box: PostgreSQL on one machine takes about 2 hours for 1 TB, spilling the join and sort to disk on a single node, while Spark on 100 machines finishes in about 3 minutes and scales linearly

Figure 3. Spark SQL runs the same SQL as Postgres but executes it distributed. The Spotify genre-by-week analytics query (a join across listens and songs) is identical text; Catalyst compiles it to RDD operations: broadcast the smaller songs table to every node, join it locally against each listens partition, then group and count. PostgreSQL on one machine takes hours at 1 TB, spilling the join and sort to disk on a single node with no cluster to parallelize across, while Spark on 100 machines finishes in minutes and scales linearly with cluster size. No code changes between single-node and distributed: the optimizer picks join strategies and partition pruning the same way a single-node optimizer does.

When Spark vs. When MapReduce

Spark wins for:

  • Iterative algorithms. Machine learning, graph processing, anything that touches the same data repeatedly.

  • Interactive analytics. Cached intermediates make follow-up queries return in seconds.

  • Multi-step pipelines. One end-to-end job instead of a chain of separate MR jobs.

  • Mixed workloads. Batch + streaming + ML in one framework.

MapReduce still wins for:

  • Datasets larger than cluster memory. Spark spills to disk when it has to, but if every stage spills, MR's design is just a better fit.

  • Single-pass batch jobs. A nightly ETL that reads once and writes once does not benefit from Spark's in-memory pipelining.

  • Cost-sensitive clusters. RAM is more expensive than disk; if latency is not the constraint, MR is cheaper.

Spark at Real-World Scale: Netflix Recommendations

# Simplified Spark ML pipeline
user_ratings = spark.read.table("user_ratings")     # 1 TB
movie_features = spark.read.table("movie_features") # 100 GB

# Traditional MapReduce would require multiple separate jobs.
# Spark expresses it as one in-memory pipeline:
recommendations = user_ratings \
    .join(movie_features, "movie_id") \
    .groupBy("user_id") \
    .agg(collect_list("rating").alias("ratings")) \
    .withColumn("predictions", ml_model_udf("ratings"))

# Cache for interactive exploration
recommendations.cache()
recommendations.show()                  # first computation
recommendations.filter(...).show()      # uses cached data

Numbers from Netflix's production reports:

  • Data: 1 TB user ratings, 100M+ users

  • Wall clock: ~15 minutes (vs. 3+ hours on MapReduce)

  • Workflow: data scientists iterate interactively instead of waiting overnight

Key Takeaways

  1. Same primitives, better execution. Spark uses the same hash partitioning and worker model as MapReduce. The change is keeping intermediate data in RAM and tracking lineage.

  2. Memory-centric design unlocks new workloads. Iterative ML, interactive analytics, and large-scale graph processing become tractable when intermediate state lives in memory.

  3. One framework, many APIs. Spark Core (RDDs) → Spark SQL (DataFrames + the Catalyst optimizer) → Spark Streaming → MLlib. Same execution engine, different surfaces.

  4. Determinism enables lineage. Spark's fault-tolerance trick only works because every transformation is deterministic. Replay produces the same result.

Next

BigQuery → Spark is the workhorse for ELT-style transformations: you take raw data, run a multi-stage in-memory pipeline, and write the result somewhere. The next case study is what to query that result with. BigQuery is the serverless OLAP engine that pairs naturally with Spark in the modern data stack.