Apache Spark: MapReduce Evolved

Concept. Spark generalizes MapReduce by representing a dataset as a Resilient Distributed Dataset (RDD): a partitioned, lazily-evaluated, lineage-tracked collection. You chain many transformations into a single in-memory pipeline that recomputes lost partitions from lineage instead of re-reading from disk.

Intuition. When a data scientist explores last year's Listens with filter(genre == 'pop').groupBy(user_id).count().orderBy(desc('count')).limit(100), Spark builds a single in-memory pipeline across thousands of cores, holds intermediate results in RAM (so the next exploratory query takes seconds, not minutes), and if a machine dies mid-job, Spark recomputes only that machine's partition from the lineage.

RDDs: The In-Memory Pipeline

Spark's core abstraction is the RDD, a Resilient Distributed Dataset. An RDD is the data as it stands after one step, a partial result on the way to the answer. Chain four transformations and you have made four RDDs, three of which are scaffolding. Each one is split into partitions across the cluster and held in memory. Spark records the chain as a lineage graph and evaluates it lazily, in one pass, only when an action finally asks for a result. So intermediate data stays in RAM between stages instead of round-tripping to disk, which is why the same three-stage pipeline (filter to pop, count by user, sort the top) runs about 100× faster than MapReduce.

Four boxes left to right joined by labelled arrows: RDD0 is the Listens table with nine rows as they arrive; a filter arrow leads to RDD1, only pop listens; a groupBy arrow leads to RDD2, one row per user showing Mickey, Minnie and Daffy, with Pluto absent because Pluto has no listens; an orderBy arrow leads to RDD3, the same three users with most plays first.

Figure 1. One query, four RDDs. Every step leaves behind the data as it stands so far, and only the last box is the answer anybody asked for. The other three are working state. MapReduce wrote each of them to disk so the next step could read it back; Spark keeps them in memory. Pluto never appears after the grouping, because Pluto has no listens.

Two panels: MapReduce runs a three-stage pipeline as three chained jobs, each writing its output to pink HDFS before the next job reads it, in about 3 minutes; Spark chains the same three stages into one lazy in-memory pipeline that keeps RDDs in pink RAM with zero disk writes in about 2 seconds

Figure 2. The same three-stage analytics pipeline (filter to pop, count by user, sort the top), two engines. MapReduce runs it as three chained jobs and materializes each job's output to HDFS before the next reads it (about 3 minutes); Spark chains the same three stages but keeps intermediate RDDs in memory and evaluates lazily, building a plan it runs once when an action triggers it (about 2 seconds). The ~100× gap comes from data staying in RAM between stages, operator fusion from lazy evaluation, and cheap lineage-based recovery instead of replication.

Resilience: Lineage, Not Replicas

The R in RDD is resilient. RAM is volatile, so if a node crashes mid-job Spark could re-read from disk (slow) or keep three copies of every intermediate (expensive). It keeps the recipe instead: the lineage, which is enough to rebuild any partition on demand.

Four RDD stages with grey partitions, all narrow transformations; the worker holding RDD3.P1 dies so that partition is pink, and Spark replays filter then map then map on just the still-alive upstream RDD0.P1 along the pink path, with no replicas stored

Figure 3. Spark recovers lost data by replaying lineage instead of keeping replicas. Each RDD records the transformation that produced it (filter, map, map), so Spark stores the operator graph, not the intermediate data. These are all narrow transformations, so each output partition depends on exactly one upstream partition: when a worker dies and takes RDD3.P1 with it, Spark re-runs filter → map → map on just the still-alive RDD0.P1. Storage drops from 3× to 1×, and the cost of a failure rises from reading a replica to recomputing one lost partition. A wide transformation like groupBy would instead re-read every upstream partition through a shuffle.

This works only because Spark's transformations are deterministic: same input, same operator, same output, so replaying the lineage reproduces the lost partition exactly.

Spark SQL: DataFrames + Catalyst

Most teams write SQL or DataFrame transformations rather than low-level RDD code, and Spark's Catalyst optimizer compiles them into the underlying RDD operations.

The Spotify genre-by-week join query on top, Catalyst's grey execution plan lower-left, and a pink box: PostgreSQL on one machine takes about 2 hours for 1 TB, spilling the join and sort to disk on a single node, while Spark on 100 machines finishes in about 3 minutes and scales linearly

Figure 4. Spark SQL runs the same SQL as Postgres but executes it distributed. The Spotify genre-by-week analytics query (a join across listens and songs) is identical text; Catalyst compiles it to RDD operations: broadcast the smaller songs table to every node, join it locally against each listens partition, then group and count. PostgreSQL on one machine takes hours at 1 TB, spilling the join and sort to disk on a single node with no cluster to parallelize across, while Spark on 100 machines finishes in minutes and scales linearly with cluster size. No code changes between single-node and distributed: the optimizer picks join strategies and partition pruning the same way a single-node optimizer does.

When Spark vs. When MapReduce

Spark wins for:

  • Iterative algorithms. Machine learning, graph processing, anything that touches the same data repeatedly.

  • Interactive analytics. Cached intermediates make follow-up queries return in seconds.

  • Multi-step pipelines. One end-to-end job instead of a chain of separate MR jobs.

  • Mixed workloads. Batch + streaming + ML in one framework.

MapReduce still wins for:

  • Datasets larger than cluster memory. Spark spills to disk when it has to, but if every stage spills, MR's design is the better fit.

  • Single-pass batch jobs. A nightly ETL reads once and writes once, and one pass is exactly the shape MapReduce was built for.

  • Cost-sensitive clusters. RAM is more expensive than disk; when throughput matters more than latency, MR is cheaper.

Spark at Real-World Scale: Netflix Recommendations

Netflix's recommendation pipeline is a join, a grouping and an aggregate. Here it is twice, once in SQL and once as a DataFrame chain, so you can see they are the same query.

The same query on two dark code cards. On the left, SQL: SELECT r.user_id and AVG(r.rating) AS avg_rating, FROM user_ratings r, JOIN movie_features f ON r.movie_id = f.movie_id, GROUP BY r.user_id. On the right, the DataFrame chain: user_ratings.join(movie_features, movie_id).groupBy(user_id).agg(avg(rating).alias(avg_rating)). Four highlights walk both panes together

Figure 5. One query, two surfaces. Read the highlights across: the table, the join, the grouping and the average are the same four operations either way, and Catalyst compiles both to the same plan, so you can write whichever one you prefer. The one real difference is order. SQL puts the average near the top, because it states the result you want; the chain puts it last, because it states the steps in the order they run. Both are lazy, so an action such as show() asks for the rows.

AVG is one aggregate among many. Spark also has array aggregates, and collect_list(rating) is the one you will meet most: instead of reducing a group to a single number, it gathers every value in the group into a list, so one row per user can carry that user's whole rating history. The next step then gets the raw values rather than a summary, which is what handing a group to a model needs.

At the scale this runs:

  • Data: on the order of 1 TB of ratings across 100M+ users

  • Wall clock: minutes rather than the hours the same work took as chained MapReduce jobs

  • Workflow: data scientists iterate interactively instead of waiting overnight

When the Map Step Is a Model Call

Figure 5 was all data work: a join, a grouping, an aggregate. Now put a model in that same chain.

The map in MapReduce was a function you wrote: given a row, return a value. That function can live anywhere, including behind a network call. Databricks' AI Functions put a language model in that slot, and the SQL stays ordinary:

SELECT song_id, title,
       ai_classify(title, ARRAY('workout', 'study', 'party', 'sleep')) AS mood
FROM songs;

One model call per row, a hundred million rows. The driver decides how many calls are in flight and pushes them out across the executors; parallelism, retries and scaling are handled the same way they are for any other stage. None of that changes the engine: the table is partitioned, the plan is lazy, a straggler is still a straggler.

A Songs table in four partitions feeds one map slot. The slot first holds a plain deterministic function, upper of title, computed on the worker; then the same slot holds a model call, ai_classify, which sends each row to a model endpoint. The executor lane below is identical either way.

Figure 6. The slot that held upper(title) now holds a model endpoint, and the engine underneath is unchanged: the same four partitions, the same executors, the same lazy plan and the same retries. Only the function moved off the worker and onto an endpoint. At this scale the hard parts are the ones this module already covers: partitioning, retries, stragglers and cost. The model is the easy part.

One row, or the whole table

Most of these functions are scalar: one row in, one value out, the map step. Three of them read many rows at once, and rows meet in a shuffle.

One row in, one value out

ai_classify · ai_extract · ai_query

Each labels, pulls fields from, or prompts on a single row. This is the map step, so it scales by adding machines.

Many rows in, one answer

ai_forecast · ai_top_drivers · ai_search

One extends a whole time series, one compares two groups, one reads a document set. Each waits on a shuffle, so the shuffle sets the limit.

The open-source engine moved the same way. Spark 4.2 added vector distance and similarity functions and a NEAREST BY clause for top-K ranking joins, which belongs on the shuffle side too: the nearest-neighbour search behind vector indexes becomes a distributed join you write in SQL.

What Breaks: Lineage Meets Non-Determinism

Figure 3 rests on one sentence: replaying the lineage reproduces the lost partition exactly. filter and map keep that promise. A model breaks it.

Ask a model to label the same song twice and you can get two answers. So when a worker dies and Spark rebuilds its partition from lineage, the rebuilt rows can differ from the ones they replaced. The job still reports success. Two rows with identical input can end up with different labels, and which rows differ comes down to which machine happened to fail.

Two tracks over the same lineage. With a deterministic function the replayed partition returns the identical value, marked with a green tick. With a model call the first run returns workout and the replay returns party, marked with a red cross, so the recovered partition disagrees with the one it replaced. Three fixes follow: pin the sampling, cache on a hash of the input row, and materialise the output to a table.

Figure 7. The same recovery that makes Spark cheap makes a model pipeline inconsistent. Production pipelines close the gap three ways, and only the last one closes it completely: temperature zero narrows the spread while the token can still move; caching the answer against a hash of the input row makes a replay read rather than ask; and materialising inference output to a table removes recomputation entirely, so real pipelines write predictions down once and read them after that.

Determinism was a footnote as long as the map step was arithmetic. Put a model in that slot and it becomes the design constraint.

Key Takeaways

  1. Same primitives, better execution. Spark uses the same hash partitioning and worker model as MapReduce. The change is keeping intermediate data in RAM and tracking lineage.

  2. Memory-centric design unlocks new workloads. Iterative ML, interactive analytics, and large-scale graph processing become tractable when intermediate state lives in memory.

  3. One framework, many APIs. Spark Core (RDDs) → Spark SQL (DataFrames + the Catalyst optimizer) → Spark Streaming → MLlib. Same execution engine, different surfaces.

  4. Determinism enables lineage. Spark's fault-tolerance trick only works because every transformation is deterministic. Replay produces the same result, and a model in the map slot breaks that promise.