Why Learn SQL When an LLM Can Draft It?

Concept. The model writes the SQL in a second. Deciding whether it is the query you meant is still your job, and a better model cannot do it for you, because the ambiguity sits in the question.

Intuition. Ask for the users who don't listen to Taylor Swift. Does a collaboration count? Does one play count, or only a favourite? Does Pluto, who played nothing, count? Two times four times two is sixteen queries, every one of them runs, and they do not agree.

You asked in English. The model wrote SQL. It ran, and it returned rows. A query that runs has proved the syntax parsed, and nothing in the output speaks to whether it answered your question.

One English Sentence Hides Sixteen Queries

The sentence leaves three choices open. You did not make them, so the model will. How do you match "Taylor Swift": the exact artist, or collaborations too? What counts as a listen: any play, a repeat, a recent play, a favourite? And does a user who has played nothing at all count as not listening to her?

One English sentence, find users who don't listen to Taylor Swift, hides three independent decisions that multiply to 2 times 4 times 2 equals 16 valid queries, drawn as a grid of 16 cells that all run. Two are labelled to show how far apart the answers are: one returns just Pluto, another returns all four. An orange ring marks the square the model landed on this run; the pick can move on another run or another model.

Figure 1. Sixteen valid queries behind one sentence. They do not agree: one returns a single user, another returns all four. Nothing in the grid marks which one you wanted, because nothing in the question does. The model picks a square (the orange ring) and hands you the SQL, so which one it picked is readable, if you can read SQL. Whether it is the one you meant is a separate question, and asking again tomorrow, on a newer model, or from a teammate’s seat can move the square. And this is the smallest schema you will ever see.

The Benchmark Says 80%. A Real Warehouse Says 30%.

Two benchmarks measure this. The distance between them is this whole module.

BIRD, the industry-standard text-to-SQL benchmark. Competitors include OpenAI, Anthropic, Google, Databricks, Snowflake, Amazon, Alibaba, Tencent, ByteDance, and Huawei. It has 12,751 questions over 95 real databases (33 GB) across 37 domains. The best systems clear about 80% execution accuracy; humans about 93%. Every top score is handed the definitions each question needs.

Figure 2. BIRD uses 12,751 questions over 95 real databases, 33 GB, across 37 domains. The best systems reach about 80% execution accuracy; humans reach 93%. Each question includes an evidence string that supplies the needed definitions. The score measures SQL generation given a supplied meaning.

That is a fair question, and 80% answers it: given the meaning, a model can write the SQL.

BEAVER asks the other half of the question. Built by Peter Baile Chen, Michael Stonebraker and colleagues at MIT from three private enterprise warehouses, its SQL comes from production query logs and its questions were written alongside the DBAs who maintain them: 9,128 queries over 812 tables across 19 domains. Nothing hands the model the meaning, because in a real warehouse nothing does.

The best agent frameworks on a current frontier model reach 10.8%. Give them perfect knowledge of every subtask (the right tables, the join keys, what each column means, the domain knowledge) and they reach 30.1%.

The model never changed between those two numbers. Structure over the data moved them, and nothing else did. The gap between the benchmarks measures who supplies the meaning.

What Has to Carry the Meaning

Something more precise than English does: a function that is verified and kept. Anthropic wrote it down as company infrastructure and went from 21% to 95% with no change to the model. What they built, and what it means for your data, is the next page.