LLMs and SQL: Syntax and Semantics

Concept. LLMs solved SQL syntax. They have not solved semantics: whether the query means what you asked. Closing that gap is your job, and the work is SQL.

Intuition. A model writes SQL that compiles and returns rows. That says nothing about whether it answered your question. To know, you read the query, run it, and check the rows, which is exactly what you cannot do to an English sentence.

Query Equivalence showed a query can match today's rows and still be wrong on other input. An LLM gives you that query: it runs, and you judge it on meaning, not on whether it fits the rows in front of you.

Syntax Is Solved, Semantics Isn't

The BIRD benchmark measures text-to-SQL. The same model that nails syntax misses meaning:

95%+

Syntax accuracy

16–77%

Semantic accuracy

runs ≠ correct

The gap

A horizontal accuracy axis. One-shot, a single blind attempt, lands near 16%. Multi-shot, the run-read-error-retry loop, reaches about 77%; the blue band between them is the lift, from method not a bigger model. The grey gap from 77% to 100% is the distance to correct that more retries cannot close.

Figure 1. The same model spans 16% to 77% on BIRD semantics. One blind attempt (one-shot) lands at the bottom; the run-read-error-retry loop (multi-shot) reaches the top of the blue band, from method, not a bigger model. The grey gap to 100% is what more retries cannot close, and what the three steps close.

The two ends of that range are two ways of using the same model:

  • One-shot: the model writes the query once, blind, with no feedback. Scores near 16%.

  • Multi-shot: the model runs the query, reads the error or the rows, and rewrites, looping until it works. Reaches about 77%.

Same model at both ends. The retry loop lifts the score, and it works only because SQL runs: each attempt errors or returns rows, and the model fixes the next from that signal. An English sentence never runs, so rewording it never tells you it was wrong. The model also never sees your data, so it guesses your domain logic unless you state it. Even multi-shot stops short of correct, because running a query for the wrong question just returns the wrong rows faster.

Browse one real BIRD question and its gold SQL to see how much a single request has to pin down:

Three Steps

The model does one thing: generate the query. You add three steps around it, each catching a different failure.

The model generates the SQL; three steps (blue) wrap that generation, each catching a different failure. Precision pins the spec and catches the wrong reading. Execution runs it on real data and catches crashes and empty results. Verification tests the result and catches subtle errors and drift. The model only generates; the three steps are yours, and all three are SQL.

Figure 2. The model generates the SQL; three steps around it catch three failures. Precision pins the spec and catches the wrong reading. Execution runs it on real data and catches crashes and empty results. Verification tests the result and catches subtle errors and drift. The model generates; the three steps are yours, and all three are SQL.

Precision, execution, verification, in that order. The next three pages take them one at a time.

Next

Precision: Define the Answer → First step: pin the meaning so the model writes the right query, worked through one ambiguous question and the queries it allows.