Why Learn SQL When an LLM Can Draft It?
Concept. An LLM can draft SQL quickly. You still have to define what the question means. Executable SQL can encode the wrong interpretation.
Intuition. Ask for the users who don't listen to Taylor Swift. Does a collaboration count? Does one play count, or only a favorite? Does Pluto, who played nothing, count? Two times four times two produces sixteen queries. Every one runs, and they do not agree.
One English Sentence Hides Sixteen Queries
The sentence leaves three choices open: how to match Taylor Swift, what counts as a listen, and whether a user with no plays counts as a non-listener.
This is the simplest example in our data, where LLMs can make up several open choices on your behalf. In production data systems (below), the missing context lead to wrong results.
Figure: One English question hides sixteen valid SQL queries, and different queries return different answersLeft: one English sentence, find users who do not listen to Taylor Swift. Middle: three independent decisions the sentence leaves open (how to match Taylor Swift, what counts as a listen, and whether a user who plays nothing counts), multiplying to 2 times 4 times 2 equals 16. Right: a grid of 16 query cells, every one a valid query that runs and returns some answer. Two are labelled to show how far apart the answers are: one returns just Pluto, another returns all four users. An orange ring marks the square the model happened to land on this run; its pick is not fixed and can move to another square on another run or a newer model. Color key: grey is a valid query, all sixteen of which run, and orange is the one the model happened to pick.
Figure 1. Three independent choices produce sixteen executable queries. One returns only Pluto. Another returns all four users. The SQL defines which interpretation the model chooses. The English request is vague about what the user intends.
What BIRD and BEAVER Measure
Figure: BIRD and BEAVER text-to-SQL benchmark settingsBIRD runs 12,751 questions over 95 real databases, 33 GB of schemas, across 37 domains. Its leading systems reach about 80 percent with evidence supplied. BEAVER uses private enterprise warehouses. Its best agent frameworks reach 10.8 percent in the standard setting and 30.1 percent when all subtask hints are supplied.
Figure 2. BIRD has 12,751 questions over 95 real databases (33 GB, 37 domains), and each question comes with an evidence string that supplies the definitions it needs: the best systems reach about 80%, humans 93%. BEAVER has 9,128 queries over 812 tables from three private enterprise warehouses, with SQL from production query logs, and withholds those definitions: the best agent frameworks on a frontier model reach 10.8%. Give the same model perfect knowledge of every subtask (the right tables, join keys, column mappings, and domain knowledge) and it reaches 30.1%. The model is unchanged; the missing structure is the gap.
The Missing Definitions
Both benchmarks expose the same requirement: business definitions that connect an English request to the intended query. Anthropic built those definitions into its analytics stack.