Project 2 slides

Project 2 overview: NanoMem and NanoQuery. Build, measure, and plan ahead.

The 10× data student

The 10× data student Build one of two systems, NanoQuery (text-to-SQL) or NanoMem (agent memory). Both share the same arc: a foundations stage you build in stages with CA support that protects your grade, then a frontier stage where your own experiments earn credit even when the idea does not work, with extra points for the top five in each track. Build one of two systems: NanoQuery or NanoMem. Both are a frontier. NanoQuery text-to-SQL engine NanoMem agent memory system Both follow the same arc: Foundations The working system, built in stages with CA support. Finish it, and your grade is protected. to the other side Frontier Your own work, past the foundation. Credit for the experiments, even if the idea doesn’t work. Top 5 in each track earn points.

How agent memory works

Local memory: capture, store, retrieve Local agent memory. A model captures records into one embedded database on your machine (claude-mem, SQLite). It retrieves selected records back into context two ways: keyword search with FTS5 finds exact terms, and an optional Chroma vector index finds related meanings. Selected records return to a later model context. 1 Model LLM + context window captures records sessions, observations, summaries One embedded database on your machine · claude-mem write · capture read · retrieve + rank 2 Keyword SQLite · FTS5 finds exact terms 3 Vector Chroma (optional) finds related meanings Retrieved records become context for a later turn; the model still has to use them well.

Your version: NanoMem

From a SQLite baseline to your own NanoMem engine Phase 0 uses SQLite FTS5 and sqlite-vec to establish baseline results. A dotted vertical line separates it from Phases 1 through 6: implement a query planner, retrieval, ranking, and results, then benchmark against SQLite. Phase 0: SQLite baseline Run the supplied prototype SQLite FTS5 + sqlite-vec Inspect search results Record baseline benchmarks Keep SQLite for comparison Phases 1–6: Build your own NanoMem Implement indexes and retrieval, then evaluate your engine Query + plannerInspect the queryChoose a method RetrieveBM25: keywordsLSH: vectorsHybrid: both RankBM25 / cosineRRF / custom fusion Return resultsDocument IDsScores + text Compare five methods against the SQLite baseline on the same data and queries. BM25 · LSH · RRF hybrid · Custom fusion · Planner-routed retrieval Measure recall, MRR@10, latency, and memory; explain your findings.

How Anthropic grounds SQL

Anthropic's analytics stack A skill routes Claude to roughly thirty governed domain files and the semantic layer before Claude writes and retries SQL against the warehouse. Evals gate each domain at about ninety percent. Governed sources are maintained across questions, and eval failures drive fixes to them. Without skills accuracy was at most twenty-one percent; raw query history moved it under one point; with skills it rose above ninety-five percent, and about ninety-nine percent in some domains. Claude writes the SQL uses governed context 1 Their warehouse queries and retries against the real thing runs and retries on its own 2 The skill routes to ~30 governed docs semantic layer first 3 Evals one suite per domain gate launch at ~90% 4 Governed sources semantic layer + reference docs human-owned definitions, tables, joins, and filters, updated as data and logic change loaded as context failures fix sources Without skills, accuracy was at most 21%; raw query history moved it under 1 point; with skills it rose above 95%, and about 99% in some domains.

Your version: NanoQuery

NanoQuery project roadmapPhase 0 prepares the data and proposal. T80 completes analysis, T87 builds a SQL library, T93 establishes ground truth, and T100 compares two text-to-SQL prompts against the answer key. Each stage links to its requirements.Phase 0: Proposal · 10 PTSTables + documents → keyword join + chart → proposalUse your Project 1 tables; prepare Wikipedia or custom domain documents with Spark and check their mappings.After Phase 0: Complete T80, then extend your workT80: Analyze · +70 PTSEmbed and retrieve passagesFive SQL queries; three chartsCheck evidence, costs, and findingsT87: SQL libraryReuse the five analytical queriesDefine parameters and domain rulesTest each definitionT93: Ground truthChoose five new questionsReuse or combine the libraryWrite SQL; verify expected resultsPhase 0 (10 PTS) + core analysis (70 PTS) = T80 (80 total).T100: Text-to-SQL experimentSame five questions, schema, output columns, retrieved IDs, model, and settings.A: Base contextQuestion + schema + IDsB: Base + T87 libraryEverything in AAdd the saved SQL libraryGenerate + run SQLSave original SQLSave results or errorsGenerate + run SQLSame execution checksSave results or errorsCompare with ground truthUse T93 reference resultsReport A: __/5 and B: __/5Explain errors and differencesKeep reference SQL and expected results out of both prompts.

Start this week

Start this weekPhase 0: October 30 · No late daysRead the project and run the starter notebook.Choose your data, partner, and target tier.Submit a working starting point and a plan.Most of the work comes after Phase 0.NanoMem projectNanoQuery project

Plan ahead

Plan aheadFinal: November 20 · 11:59 pm Pacific5 sections and lots of office hours per week, over the next 5 weeks.November 18–20: private Ed questions only.Staff clarifications end November 20 at 6 pm.One submission per team.Late days are the minimum of what you and your partner have.Plan ahead.Full late-day policy