Execution: Run It on Real Data

Concept. Step two: execution. A one-shot query has no feedback, so a wrong table name or a bad join just fails, and the model never learns. Give the model an execution environment, a real database it reaches through a tool like MCP or a CLI, and it writes, runs, reads the error, and corrects. That loop is the difference between one-shot and multi-shot, and it explains much of the multi-shot gain on benchmarks like BIRD.

Intuition. It is the loop you run by hand: write a query, run it, read the error, fix it. The model just does it in seconds, many times over, until the query runs.

You defined the spec on the last page. Now the model writes SQL against it. A single attempt rarely lands: the first query references a column that moved, or joins on the wrong key. With nowhere to run it, the model ships that query blind. With a database to run against, it iterates.

The Loop

An English question goes to an LLM, which takes one of two paths. Grey: one blind shot with no feedback, so it may crash, be empty, or be wrong. Blue: with a database through a tool like MCP or a CLI, it writes, runs, reads the error, and fixes, repeating until a guess becomes a query that runs. The output is a query that runs, though running is not yet right.

Figure 1. A one-shot model emits SQL blind (grey) and ships whatever it wrote. Wired to a database through MCP (Model Context Protocol) or a CLI, it runs the query, reads the error, and corrects (blue), so a guess becomes a query that runs. The loop is only possible because it is SQL: you cannot run an English sentence. This is multi-shot.

What It Catches, What It Misses

Execution catches mechanical errors for free (a missing column, a type mismatch, a join that returns nothing) and the model fixes them; it cannot catch a query that runs cleanly but counts the wrong thing, like a guessed definition of active users.

Execution catches everything mechanical, for free, because the database does the checking: a missing column, a type mismatch, a join that returns nothing. The model sees the error and fixes it without you.

It cannot catch a query that runs cleanly and returns a plausible but wrong number. The database has no opinion about what you meant; it runs what it is given. You defined the meaning with the spec, but runs ≠ correct. Verifying that is the next step.

Key Takeaways

  1. One-shot has no feedback; multi-shot iterates. An execution environment turns a single guess into a write-run-read-correct loop.

  2. Execution catches mechanical failures. Crashes, type errors, and empty results all surface the moment the database runs the query.

  3. It cannot catch a wrong-but-running query. The database has no view on your intent. That gap is the next step, verification.

Next

Verification: Prove the Result → The model iterated to a query that runs. But runs ≠ correct. Third step: verification. Test the result, the way software trusts code, with unit tests and regression.