Case Study 1.1: Anthropic's Self-Service Analytics Stack

Concept. English is too vague to hand a model, so somebody has to write the meaning down somewhere precise. Anthropic built that as company infrastructure, and every point of accuracy they gained came from that infrastructure, with the model unchanged. On your own data the same parts are a table-valued function, a unit test, and a library.

Intuition. Define listener of Taylor Swift once as listener_of('Taylor Swift'), check it leaves Pluto out, and keep it. The model stops guessing what a listen is and calls the term instead.

Case Study 1.1 Reading Time: 5 mins

Ask Claude, inside Anthropic, for last week's active users. The bare model is right 21% of the time. It can already run SQL against the warehouse and retry on its errors, so the mechanical failures are gone. What it cannot do is know what active user means inside Anthropic, so it guesses.

Four things later, the same model is right 95% of the time.

What They Actually Built

Four parts around a model that only generates, drawn as two tiers. The top tier is what happens for one question: a skill that routes the model to the roughly thirty documents defining that domain, then Claude in grey which only generates, then their warehouse which the model queries and retries against, then a suite of evals. The bare model already had the warehouse and scored twenty one percent. The bottom tier is the governed semantic layer, lower and wider as a base that only grows: whatever clears the evals becomes a definition, and an arrow runs from the layer back up into the skill.

Figure 1. Four parts around a model that only generates. Their warehouse is part one, and the bare model already had it, which is why 21% is the honest starting number. The skill that loads a domain's ~30 definition docs is part two, and it is the mandatory default path. Evals are part three, and a domain reaches production only once it clears ~90%. The semantic layer is the lower tier: whatever clears the evals becomes a definition in it, and it feeds back up as the context for the next question.

You will hear this diagram called other things. Supplying retrieved context to a model is what the industry calls RAG, retrieval-augmented generation; "grounding", "curation" and "context engineering" name the same loop from different angles. The label changes; the loop does not.

What Actually Moved the Number

What the model was givenAccuracy
Anthropic's warehouse, nothing else21%
Every query the company had ever run21%, plus less than a point
A governed semantic layer, gated by evals~95%, and ~99% in some domains

The middle line is the one that matters. Before building any of this, they tried the obvious shortcut: hand the model the entire query history, every query anyone had ever written against that warehouse. It moved accuracy less than a single point.

Access to the data was never the missing piece. The loop was.

These are Anthropic's internal numbers: their own warehouse, their own questions, their own evals, written up in "How Anthropic enables self-service data analytics with Claude". They are not BIRD, the public benchmark, and 95% on your own warehouse is a narrower test than a leaderboard.

How to Start on Your Own Data

You already have the warehouse: it is the Postgres or BigQuery you have been querying since Module 1. The rest of the loop is three moves you can start today. It begins with one decision.

Put the meaning in a function, not a prompt

The ambiguity has to be written down somewhere, and you get two choices. Spell every decision out in the prompt and the model has one reading to pick, which answers that question and nothing else; ask a slightly different one tomorrow and you write the whole spec again. Or operationalize the phrase carrying the ambiguity, listener of Taylor Swift, so it is written once and every later question calls it.

The vague question, users who don't listen to Taylor Swift, with three decisions left open and sixteen queries, forking two ways. The top branch in grey spells every decision out in the prompt: it answers this one question, and the next question means writing the whole spec again. The bottom branch in blue operationalizes the term instead, as three numbered steps: define it once, call it by name, set up a unit test.

Figure 2. Both branches are precise. Only one of them you do once. The same three decisions get settled either way; the difference is whether they survive the question that settled them. The three numbered steps are the whole job, and the next section does each one in SQL.

Write it as a table-valued function

This is the SQL library from Module 1, doing a new job. Nothing about writing it changes: define the term, call it by name, set up a unit test.

Three stacked panels. First, in blue, the definition: listener_of created once as an ordinary SQL function taking an artist and returning the set of user ids who listened. Second, the usage: the original question rewritten to call listener_of by name instead of spelling the joins out again. Third, in green, the unit test: calling listener_of for Taylor Swift returns Mickey, Minnie and Daffy, and Pluto, who played nothing, is absent.

Figure 3. The definition fixes decisions that later queries would otherwise repeat. DISTINCT chooses how repeat plays count. The JOIN chooses what a listen means. A user with no listens never appears, so Pluto does not qualify as a listener. STABLE lets Postgres inline the function; without it Postgres plans the function as a black box and rebuilds it on every call. Later queries compose functions, and the model calls the term instead of inventing a definition.

Fence the queries your library does not cover

Composing defined terms keeps the agent on rails most of the time. When no term fits it falls back to raw SQL, and there it can join tables that should never touch. Users.user_id = Songs.song_id is two integers, so it runs, and it hands you a confident wrong number.

Do not try to enumerate the bad joins; there are more of them than you can list. Write the allow-list instead: a join is allowed when it follows a foreign key your DDL already declares. In the Spotify schema that is Listens to Users on user_id and Listens to Songs on song_id, and that is the whole list. Anything else, the agent escalates to you rather than guessing.

That is a small set, and small is the point. A short allow-list you can check beats a long forbidden-list you cannot finish.

Then it goes in the library

Keep a unit test for each decision the term settles, and trace it with a debug table if a result surprises you. Then it goes in the library, and the library is what you hand the model next time.

The next term is the same three beats. active_since(days) settles what counts as recent, top_genre_of(user) settles what to do with a tie, repeat_listener_of(artist) settles how many plays is a repeat. Each one is a small argument you have once, and the library is where the answer stays. That is Anthropic's semantic layer, scaled down to your schema.

What you end up owning

A timeline in three moments. Along the top, in grey, the model is replaced each time: a new version lands every few weeks and swapping it is a configuration change. Along the bottom, in blue, the library you own grows instead of being replaced: three functions with seven tests, then nine with twenty one, then twenty two with fifty four. A panel lists what carries over unchanged: the database it runs on, the specs, the tests, and the named functions. Nothing in the library is re-verified on a swap; the suite is what every new model has to pass.

Figure 4. The model is rented; the library is owned. New versions land every few weeks and swapping one is a config change. Nothing underneath is re-verified, because none of what makes an answer trustworthy depends on which model produced it.

Takeaway: 21% to 95% on Anthropic's own data, with no change to the model. Every point came from the parts a model cannot supply for itself: the definitions, the checks, and the layer that keeps what passed. You build those with the SQL you already have. Next, the substrate underneath all of it: how ChatGPT stores its chats.