Compression Basics: Making Data Smaller
Concept. Columnar compression (run-length, dictionary, delta) exploits repetition inside one column, typically shrinking on-disk size 10× thereby cutting IO by the same factor.
Intuition. Mickey appears in 3 of 9 Listens rows. At a billion rows, his user_id might appear 100 million times contiguously. Run-length encoding stores "Mickey × 100M" instead of repeating the string, Listens shrinks 10× and reads 10× fewer pages.
When to Use What?
Figure: Four columnar compression techniques at a glanceFour panels side by side, one per technique. RLE turns a run like Mickey times three, Daffy times four into one value plus a count, 10 to 100 times smaller, very fast, best for sorted repeated values. Dictionary replaces low-cardinality strings with small integer codes, 5 to 20 times, fast, best for low-cardinality strings. Delta stores the difference between sorted numbers, 5 to 10 times, fast, best for sorted numbers and timestamps. Bit packing stores small integers in the minimum bits, 4 to 8 times, very fast, best for small integers and flags. In each panel the raw input is grey and the encoded result is orange. Color key: orange is the encoded result, grey is the raw input.
Figure 1. We can compress a column by removing repeated structure. RLE stores each run as a value plus a count (i.e., length of a run of Mickeys). Dictionary encoding maps each distinct value to a small integer code. Delta encoding stores differences between adjacent sorted numbers. Bit packing stores small integers in the minimum number of bits. The raw input is grey, the encoded result orange. Ratios run from about 4x (bit packing) to 100x (RLE on sorted data); choose based on the column's value distribution.
Run-Length Encoding (RLE)
A sorted column with long runs of the same value. Instead of storing Mickey, Mickey, Mickey, Mickey ... a hundred million times, store Mickey × 100M. One value plus a count. Compression ratios of 10× to 100× are normal on user_id when the table is physically sorted by user_id, on country in a sorted log, on any column where the same value lingers. The runs come from the data's sort order on disk, not from a query.
Dictionary Encoding
When a column has a small set of distinct values, replace each value with a small integer index into a dictionary. Genre has maybe 50 distinct values; storing each as a 1-byte code instead of a 16-byte string is a 16× win. Country, currency, status flags, all good targets.
Delta Encoding
For sorted numeric columns, store the difference between consecutive values rather than the values themselves. Timestamps arriving microseconds apart turn into tiny deltas that pack into 1-byte ints instead of 8-byte longs. Auto-increment IDs and sensor readings benefit the same way.
Combining Techniques
You run the techniques in sequence, each one on what the previous step produced, so the savings compound. This works only under the condition that makes each step work: the column is physically sorted (or clustered) by user_id and has limited cardinality. Take a 1M-row Listens user_id column sorted by user_id, with about 25,000 active users:
Figure: Stacking compression: 4 MB down to 50 KBOne user_id column of 1 million values, 4 MB raw, sorted by user_id with about 25,000 active users, shrunk in three stacked steps, drawn as a descending staircase of orange boxes whose height tracks the size. RLE collapses the sorted runs to 200 KB. Dictionary shrinks each value because 25,000 ids need only a 2-byte code, not a 4-byte int, to 100 KB. zstd is a general-purpose codec that entropy-codes the leftover byte patterns, to 50 KB. The end result is 98.75 percent smaller. Color key: orange is the column shrinking, grey is the step applied.
Figure 2. A 1M-row user_id column sorted by user_id with ~25,000 active users compresses by chaining encodings. RLE collapses the sorted runs (4 MB to 200 KB). Dictionary encoding then replaces ids with codes; 25,000 ids fit in a 2-byte code rather than a 4-byte int (to 100 KB). zstd entropy-codes the remaining byte patterns (to 50 KB), a 98.75% reduction. Unsorted data or high cardinality is less compressible.