Apache Kafka: Real-Time Event Streaming

Concept. Kafka is a durable, partitioned, replicated commit log: producers append events to per-topic partitions, consumers read them at their own pace, and the log itself is the source of truth, so multiple downstream systems can independently replay the same stream of events.

Intuition. When Mickey hits play, the Spotify client emits a play_started event to the plays topic. Kafka appends it to one of the topic's partitions (chosen by hash(user_id)), replicates it to 3 brokers, and now both the recommendation pipeline and the royalty-payout pipeline can read that exact event independently, at their own speed, replaying as far back as the retention window allows.

A Worked Example: One Play, Two Pipelines

Trace what happens when Mickey hits play on a Taylor Swift track.

  1. Produce. The client builds a play_started event and sends it to a Kafka producer, which hashes user_id to pick a partition of the plays topic and appends the event to that partition's log.

  2. Replicate. The partition lives on three brokers (1 leader, 2 followers); the leader waits for followers to acknowledge, so the event survives any single broker dying.

  3. Consume independently. The recommendation pipeline reads Mickey's event within milliseconds and updates "Up Next"; the royalty pipeline reads it later in its nightly batch. Each tracks its own offset and never blocks the other.

  4. Replay on demand. Ship a bug in the model, rewind the consumer to yesterday's offset, and re-read the same events with fixed code. The log is the durable source of truth, so the replay is bit-identical.

The producer hashes user_id to pick one of three pink topic partitions P0 to P2, each replicated 3x; two independent grey consumers read at their own offsets, the recommendation pipeline at offset 16 in real time and royalty at offset 12 nightly, and can replay any past offset

Figure 1. Kafka is a partitioned, replicated commit log. The producer hashes each event's key (here user_id) to pick a partition, so one user's events stay in order; the topic's 3 partitions each replicate 3× across brokers, giving parallelism and ordering from partitions and durability from replicas. Two independent consumers, Recommendations and Royalty, track their own offsets and read at their own pace, and because events stay until retention expires, either can rewind and re-read. Typical scale: millions of events per second, under 10 ms end to end, retained for days or longer.

Queue vs Log

People call Kafka "a message queue." It is really a log, and that one difference is why a new consumer can join later for free.

A queue consumes on read: once a consumer pops a message, it is gone, so adding a second downstream system means re-publishing everything. A log persists after read: every event sits in the log until retention expires (days, sometimes forever), so a new analytics team, ML pipeline, or fraud detector can join years later and read the entire history without disturbing existing consumers.

Delivery, in One Line

Delivery is tunable, at-most-once (may drop, never duplicate), at-least-once (never drop, may duplicate), or exactly-once (via the transactional API). In practice almost everyone runs at-least-once with idempotent consumers, and reaches for exactly-once only when money is on the line.

Where Kafka Fits in the Big Picture

Kafka is not a competitor to MapReduce or Spark, it is the connective tissue that feeds them. Producers append events, consumers read them at their own pace, and that is the whole job: a durable pipe between the systems that produce data and the systems that process it. MapReduce and Spark work on data that already exists in storage; Kafka is the piece that gets it there in the first place, in real time, and keeps it replayable.

Real-World Use

LinkedIn (where Kafka was born) runs about 1 trillion messages a day; Netflix about 4 million events a second. Most high-volume event pipelines at large internet companies run on Kafka.

Wrap-up

Across this module you walked every layer of the modern data stack: storage that survives failure (GFS, DFS), computation that scales (MapReduce, Spark), and now an event substrate that moves data in real time (Kafka). All three rest on the same primitives: hash partitioning, replication, and consensus. Next time you hear "we built it on Kafka" or "we run Spark on S3," you can decompose the architecture into those primitives.