Case Study 1.1: Anthropic's Self-Service Analytics Stack
Concept. Anthropic raised Claude's accuracy on its internal analytics evals from at most 21% to >95% by using skills that route it to governed definitions and verification. On your own data, start by defining table-valued functions, unit tests to check it, and a library to make your queries reusable.
Intuition. Define listener of Taylor Swift once as listener_of('Taylor Swift'). Test that it leaves Pluto out, then let later questions call the same definition.
What They Built
Read Anthropic's experiments.
Figure: Anthropic's analytics stackA skill routes Claude to roughly thirty governed domain files and the semantic layer before Claude writes and retries SQL against the warehouse. Evals gate each domain at about ninety percent. Governed sources are maintained across questions, and eval failures drive fixes to them. Without skills accuracy was at most twenty-one percent; raw query history moved it under one point; with skills it rose above ninety-five percent, and about ninety-nine percent in some domains.
Figure 1. A domain skill grounds Claude in domain-specific curated definitions: it must use the semantic layer first, then about 30 reference files on the domain's tables, columns, joins, and gotchas, and evals gate each domain before launch (initially at about 90%). With only the warehouse, Claude scored at most 21%. Raw retrieval access to thousands of prior queries moved that by less than a point. With domain skills, accuracy rose above 95%, and around 99% in some domains. These are Anthropic's own evals on its own warehouse (source), a narrower test than public benchmarks like BIRD and BEAVER.
How to Start on Your Own Data
On your own data, start by encoding one ambiguous term.
Keep the meaning in a function
A detailed prompt can pick for one question. A table-valued function defines them once and gives later questions a shared definition. Here, listener_of('Taylor Swift') records what counts as a listener.
Figure: Two ways to make the question preciseOn the left, the vague question, users who don't listen to Taylor Swift, with three decisions left open and sixteen queries. It forks two ways. The top branch, in grey, spells every decision out in the prompt: it answers this one question, and the next question means writing the whole spec again. The bottom branch, in blue, operationalises the term instead, shown as three numbered steps: define it once, call it by name, set up a unit test. That one is written once and answers every later question that mentions listening.
Figure 2. Both approaches make the current question precise. The function preserves the decisions for reuse. Define the term, call it by name, and test each decision it encodes.
Write the definition as a table-valued function, exactly like listener_of in SQL Libraries, and give it a unit test.
Queries Not in the Library
When no term fits, the agent falls back to raw SQL and can join unrelated columns, such as Users.user_id = Songs.song_id: the query runs and returns a wrong number. Allow only the joins the schema's foreign keys declare (Listens to Users on user_id, Listens to Songs on song_id) and have the agent escalate any other.
What You Own
Figure: What carries across model changesAn illustrative timeline. The example library grows from three verified functions and seven tests to nine functions and twenty-one tests, then twenty-two functions and fifty-four tests. Three model versions appear above it. The database, specifications, tests, and named functions carry across versions, and each replacement model must pass the existing test suite before serving questions.
Figure 3. The library grows one tested definition at a time: active_since(days) defines what counts as recent, and top_genre_of(user) defines ties. Swap the model and the database, definitions, and tests stay; the suite checks the new model before it answers questions.
Takeaway: Anthropic's gain came from domain skills that route Claude to governed definitions. On your own data, tested SQL functions play that role.