Pandas: Deconstructing Small Analytics
Pandas, Polars, and where one machine stops.
Concept. Pandas normally materializes a DataFrame in memory. Polars can use lazy and streaming execution to process supported workloads in batches. Both remain single-machine engines unless you choose a distributed backend.
Intuition. A laptop handles the small Spotify example comfortably. As the data grows, execution moves from in-RAM work to streaming, disk-based algorithms, and eventually many machines.
Take an example Spotify dataset: one million songs, fifty million listens, and a few thousand users. Three tables occupy a few GBs.
Pandas reads the data into memory before it computes.
Figure: How Pandas holds a DataFramePandas reads listens.csv from disk into memory as a DataFrame with named, typed columns. The DataFrame stores each column in a typed array and materializes its data in memory.
| user_id | song_id | rating |
|---|---|---|
| int64 | int64 | int64 |
| 1 | 12 | 5 |
| 1 | 45 | 4 |
Figure 1. A CSV on disk or a Parquet file reads into a DataFrame with named, typed columns. The system stores each column as its own typed array, which makes column aggregation fast. The DataFrame keeps every column of every row in memory at once, so it runs fast on small data and fails when the data exceeds memory.
A familiar example
The task: the top 10 songs by play count.
In pure Python you would write nested for loops over the rows, build a dictionary of counts, and sort it at the end. That is imperative code: you spell out how to compute the answer, step by step.
Pandas and Polars express work as DataFrame transformations. SQL expresses the result declaratively. The Polars and SQL versions appear side by side:
Polars (Python)
import polars as pl
popular = (
listens_df
.join(songs_df, left_on='song_id',
right_on='song_id', how='inner')
.group_by(['title', 'artist'])
.agg(pl.len().alias('plays'))
.sort('plays', descending=True)
.head(10)
)
SQL
SELECT title, artist, COUNT(*) AS plays
FROM listens
JOIN songs ON listens.song_id = songs.song_id
GROUP BY title, artist
ORDER BY plays DESC
LIMIT 10;
The code states the join, grouping, and order. The execution engine chooses how to perform them. Neither version spells out loops, counters, or a sorting algorithm.
SQL separates the requested result from the execution plan. A compatible distributed engine can preserve the query's shape while changing how it scans, joins, and aggregates the data. Scaling may still require partitioning, schema, and engine-specific changes.
Figure: Pandas: the whole dataset has to fit in RAMPandas reads a CSV or Parquet file into a DataFrame on one machine. Group, sort, and join operations use that machine's memory. As data and intermediate results approach the available RAM, the process may slow, swap, or fail an allocation.
Figure 2. Group, sort, and join execute in one process. As memory pressure grows, Pandas may slow sharply, swap, or fail an allocation. It does not automatically turn the operation into a streaming or distributed plan.
Pandas, Polars, and a database
Spotify reports a catalog of more than 100 million tracks, and many more listening events. At that scale, the workload determines whether one machine, or a distributed system fits.
Figure: How far each engine gets on a data-size axis, and where one machine endsAn illustrative data-size axis compares three execution models. Pandas normally materializes data and intermediate results in memory. Polars can stream supported operations in batches on one machine, although some operations still materialize. A distributed database divides storage and execution across machines. Actual limits depend on the query and hardware.
| One machine is enough | Needs many machines |
|---|---|
| Hardware you already own | A cluster you rent |
Figure 3. Pandas normally materializes the data and intermediate results in memory. Polars can stream supported operations in batches, which extends the range of one machine. Some operations still require materialization. A distributed engine divides storage and execution across machines when one machine no longer fits the workload.
Where this leaves Pandas in 2026
When the data fits in memory, a DataFrame on a laptop is the right tool, and the ceiling on one machine rises every year with improvements in hardware.
Data can outgrow a DataFrame before it requires a cluster. Streaming and disk-based execution cover much of that middle range.
The next case study uses a single SQLite file to give a coding agent persistent memory.
Takeaway: Pandas fits in-RAM analytics. Polars extends the range of one machine for supported workloads. Databases add durable storage, query planning, and a path to distributed execution.