What Module 6 solves: scale, architecture, and trust once one machine is not enough
Three constraints drive system design. Scale: a dataset exceeds one machine, so the system partitions data across a cluster and the engine must match the hardware limits at that scale. Architecture: a data model (relational, JSONB, vector) and a deployment choice (one system or several) determine which features applications can implement. Trust: a large dataset increases exposure, so the system must block attackers and preserve anonymity for people represented in the data.
An AI agent like OpenClaw consists of a loop and a memory.
3 figures
case-study-openclaw · An agent is a loop and a memory
An agent is a loop and a memory
OpenClaw stores memory in a folder on disk plus a derived SQLite index. The loop reads, acts, and writes each turn; each claw uses its own folder. The files are the canonical state: MEMORY.md holds curated facts loaded at session start, and daily notes store an append-only log. The dreaming pass merges retained notes into MEMORY.md and deletes the rest. agent.sqlite indexes chunked text and embeddings and can be rebuilt from the files.
Notes ↗case-study-openclaw · Share the memory, and Module 4 is back
Share one memory across claws, and Module 4 is back
Two claws write one shared MEMORY.md. Both read and both save; claw B saves last and overwrites claw A's edit. The overwrite creates a lost update. A fix applies Module 4 techniques to files: take a file lock so one writer updates at a time, or use append-only, idempotent writes so replay stays safe. Module 4, concurrency control.
Notes ↗case-study-openclaw · Files for a few agents, a database for many
For a few agents, files. For many, a database.
The agent count drives the memory design. A few agents can use separate folders: plain text, git-diffable, no conflicts. Many agents require one shared store with concurrent writes. A database (Postgres with pgvector, or Tencent's agent memory) serializes writes with Module 4 transactions, and a pipeline maintains a readable copy. Multi-agent systems and autonomous research push deployments toward the many-agent end.
db-design-startup · The First Scaling Move: Read Replicas
Read replicas: one primary for writes, many copies for reads
Writes go to the single primary (blue). Reads spread across N replicas (violet). The primary streams its changes to the replicas asynchronously (dashed), so a replica can trail the primary by milliseconds. Add replicas to increase read capacity without changing the write path.
Same SQL shape, new data types: SELECT/FROM/WHERE stays, only the function changes
Domain-specific types plug into the same SELECT / FROM / WHERE / ORDER BY shape through functions. A query can place a distance predicate in WHERE, extract a JSON path in SELECT, or call an LLM as a computed column.
At big-tech scale, the data stack splits into three systems: OLTP for user-facing writes, OLAP for analytics over billions of rows, and an ETL/ELT pipeline that moves data from one to the other.
2 figures
db-design-bigtech
Three systems, one data stack: OLTP serves users, ETL/ELT moves data, OLAP answers analytics
One workload split across three systems. OLTP (blue) serves live writes and targets sub-100 ms latency. OLAP (violet) serves analytical scans and reads billions of rows per query. Spark reshapes each day's data and Kafka ships it between the systems. The scale and cost numbers give the order of magnitude for each layer.
Notes ↗db-design-bigtech · ETL/ELT: Moving Data Between Worlds
ETL versus ELT: the same three steps, extract transform load, in a different order
ETL transforms before loading. A nightly Spark job cleans and aggregates raw data, then writes finished tables into the warehouse. ELT loads raw data into cheap storage, then transforms inside the warehouse with SQL and dbt. Kafka ships the data in both approaches.
Raw pipelines fail in recurring ways. A value goes missing, a row duplicates, a field arrives with the wrong type, an event arrives late and out of order, or an upstream rename drifts the schema. Slate marks a clean field. Red marks a corrupted value that entered the warehouse.
A failed test stops the build before the bad number ships
A failed test blocks the build. Each model runs as a `SELECT`. Each test asserts a property of the model output. `unique(listen_id)` passes (green). `not_null(genre)` fails (red) because thousands of podcast plays have no genre. The failure stops the pipeline, so the `top_genres` table never builds and the dashboard keeps yesterday's value. `NOT NULL` and `UNIQUE` constraints enforce the same properties at write time. dbt enforces them during transform.
Notes ↗data-quality · Observe what you cannot predict
Observability catches the drift the top-line hides
A global metric can stay inside its expected range while a slice breaks. Total plays remain within the expected band, so tests and dashboards stay quiet. A per-city split exposes Chicago falling outside its learned baseline (red), which triggers an alert. Uber built this approach in D3 and reduced detection time from forty-five days to two. Slate marks the metric. The grey band marks the expected range. Red marks the drifting slice.
Constraints (`NOT NULL`, `UNIQUE`) enforce properties at write time. dbt tests block failures you can name before they ship. Observability alerts on failures you cannot name by detecting drift. An orchestrator runs the sequence on a schedule and gates downstream work on these checks. Slate marks a rung. Each higher rung covers cases that bypass lower rungs.
Optional 6.3c · Big Schemas: What ELT Actually Builds
A warehouse grows by derivation.
4 figures
big-schemas · What actually gets big
One visit, three organisations
One visit produces three records in three organisations. FHIR defines dozens more resources. Claim references an Encounter. That Encounter references a Patient. Different companies maintain each record, so references cross company boundaries. The standard changes on a schedule outside your control. The source system chose the format, and the warehouse receives it in that form.
Two organisations run the same four steps without coordination. The hospital publishes visits delivered. The insurer publishes visits paid for. Both counts describe the same visits and can disagree. A denied or still-pending claim produces a visit the hospital delivered and the insurer did not pay for. Each pipeline can still satisfy its own definition. Another pipeline can reconcile the two by reading both outputs.
One build splits into stages. Raw keeps FHIR names chosen by the source. Staging flattens nested JSON, assigns types, drops duplicates, and renames columns to business terms, so raw_encounter becomes stg_visit. Intermediate joins the three parties, then prices each visit from its claim lines. Marts model business entities. Aggregates roll claims up to the member-month because per-claim granularity does not serve reporting. Metrics store named numbers. Exposures are dashboards and reports that read the metrics.
A pipeline is a dependency graph of models. Each dot is a model. Each line is one model reading another. Columns group models by stage from Figure 3. The highlighted route names each step: raw_encounter lands source data, stg_visit cleans it, int_visit_costed joins parties and prices visits, fct_visits shapes the business table, and readmission_rate_30d stores a metric. One route uses six models inside a graph of forty-two.
Optional 6.3d · Lineage: What Broke, and What Will I Break?
The graph that 6.3c produced supports two traversals.
5 figures
lineage-blast-radius · Six models, all green
Every model green, the number still wrong
Every model is green and the number is still wrong. The walk starts at the symptom and moves upstream one edge at a time until the values stop making sense. It ends at int_visit_completed, whose filter tests a status value the source no longer sends. Nothing errored, because an empty result is a valid result: the count is zero, the average is undefined, and everything above it computes on nothing. This is the failure 6.4 calls schema drift, found by walking the graph.
Notes ↗lineage-blast-radius · One graph, two directions
One graph, two directions
One graph, one node, two directions. Walking to the ancestors answers why is this number wrong; walking to the descendants answers what will I break. Debugging and change management are the same query here, with the arrows reversed. Both run from a terminal against the graph you already have.
The same change, asked at two granularities. Model-level lineage sees only that a model touched the table, so it reports the whole subtree. Column-level lineage classifies each dependency: copy passes the value straight through, so an export and a feature table need backfilling; transform computes from it, so a BMI calculation, a risk score and a regulatory measure all change their numbers; inspect only reads it in a WHERE or a JOIN, so values do not change but row counts might. Three different jobs, and the coarse answer hid all three.
Notes ↗lineage-blast-radius · Why the finer answer costs more
One edge you declared. The rest is inside the SQL.
The one edge you declared, and the three reasons the rest is work. ref() gave the tool stg_measurement feeding fct_visits for free. Everything finer is buried in the query: the star has to be expanded before anything knows what it selected, weight_lb is a name that exists only in the output, and the WHERE changes which rows survive without touching a single value. Answering what depends on this column means resolving all three.
Notes ↗lineage-blast-radius · The agent walks it too
Guessing, or querying
The same question asked twice. A model with no access to your warehouse can only produce something shaped like an answer, and a plausible list of model names is exactly the failure that is hardest to notice. With the graph the question stops being a prediction and becomes a lookup.
An LLM agent sends a long, nearly identical prefix on every turn: system prompt, tool schemas, and conversation history.
5 figures
case-study-kv-cache · Every turn is mostly the same prefix
The KV cache across agent turns
An agent resends most of its prefix every turn, so caching it saves the most. A request is a long prefix that only grows at the end (system prompt, tool schemas, then the conversation so far) plus a short new message; without caching, the model recomputes the whole prefix each turn, so cost climbs as the conversation grows (grey). Prompt caching stores that prefix's attention state and reads it back on every later turn, so each turn computes only the new message and cost stays flat (slate). It is the same trade as keeping your working set in RAM from Module 2: a cache hit is cheap, recomputing is not.
Notes ↗case-study-kv-cache · Every turn is mostly the same prefix
What the KV cache stores: the prefix as key, the model's intermediate results as value
What the KV cache stores. The key is the prefix (its tokens); the value is the model's intermediate work from reading it, a key vector and a value vector for each token, at every layer. It is the precomputed state the model would otherwise rebuild each turn, not the token text and not the output, so caching it skips the expensive part: you reuse the work instead of running the prefix through the network again.
Notes ↗case-study-kv-cache · Paging: fit many caches in one GPU
PagedAttention pages the KV cache into fixed-size blocks instead of reserving a chunk per request
R1, R2, and R3 are three chats served on one GPU at once, each holding a KV cache (the keys and values the model computed, not the prompt text) that grows one token at a time. The naive way (left) gives each request one contiguous chunk of GPU memory sized to its longest possible length, so most of it is reserved and never used, wasting 60 to 80 percent of GPU memory. PagedAttention (right) splits each request into fixed-size blocks, packs them with no gaps, and keeps a page table mapping each request to its blocks, which can sit anywhere (request one is in blocks 0 and 2, not adjacent), dropping waste under 4 percent. This is the page-table half of Module 2 paging, packing one memory tier rather than moving data between disk and RAM.
An agent keeps two kinds of memory, and one lever shrinks both. Long-term memory is the vector store it retrieves from, kept small by TurboQuant (Case Study 2.4); working memory is the KV cache of the current context, kept small by KV-cache quantization. Both are dense-vector piles too large to hold at full precision, so both use the same move: drop precision, keep the signal.
The techniques that keep an agent's KV cache small and fast are Module 2 primitives under new names. PagedAttention breaks the KV cache into fixed-size pages addressed through a page table, so scattered GPU memory packs tight instead of demanding one contiguous chunk; it is the same fixed-size-page idea as Module 2 paging, borrowing the page-table half rather than the disk-to-RAM movement. Prefix and prompt caching turn a recompute into a cheap cache hit, priced by the same cost-model reasoning as Module 2 (here the expensive operation is compute, not a disk read). KV-cache quantization is drop-precision quantization, the same lever as TurboQuant in Case Study 2.4, and offloading the cache across GPU, CPU, and disk is the storage hierarchy. Nothing in the right column is new to you.
Zero Trust assumes compromise at every layer: network, users, admins, code, and backups.
2 figures
data-security · Zero Trust: The Five Surfaces
Zero Trust: assume every surface between a request and the data is already hostile
Zero Trust treats every path to the data as hostile until verified. Attackers can sniff the network, phish a user, compromise an admin account, inject code, or encrypt backups with ransomware. Defenses verify each request at each surface and apply least privilege, MFA, and immutable backups.
Notes ↗data-security · SQL Injection: The One Bug That Keeps Winning
SQL injection: the same attacker input is a command in a concatenated query but harmless data in a parameterized one
The same input can act as SQL or as data. String concatenation lets an attacker terminate a string literal and inject a new command, so DROP TABLE users executes. A parameterized query sends values separately from SQL text, so the database treats the input as one value and executes no injected command. Parameterize every query and never concatenate input into SQL.
The linkage attack: joining anonymized Netflix data to public IMDB reviews re-identifies a user
A join on (movie, rating, date) can map an anonymous ID to a real name. A user's ratings form a rare pattern across movies, scores, and dates. The same user can publish some of those ratings on IMDB under a real name. The join links the random Netflix ID to that name and reveals the rest of the Netflix history under the same ID.
Case Study: Google COVID Reports (Differential Privacy)
Removing names does not make data anonymous.
1 figure
case-study-privacy
Differential privacy turns an exact count into a safe, noisy one
Differential privacy protects the released report. The exact query returns a true count (say 100 visits at the center Bob went to), which on its own can identify him. Differential privacy adds calibrated Laplace noise, sized by a privacy budget (ε = 1), so the published number comes back a little different each run (98, 103, 101). The trend across locations stays accurate, but Bob's single visit can never move the answer.
sql-trends · Beyond the Syntax: The Economics of Scale
Basic Postgres plus a JSONB column goes a long way, versus spinning up specialized systems
Basic Postgres plus a JSONB column supports typed columns and semi-structured fields in one row and keeps them queryable with SQL. A managed provider like AWS, GCP, Supabase, or Neon runs this setup for ten to a hundred dollars a month. Big Tech runs a hybrid stack: an OLTP database, an OLAP warehouse, an ELT pipeline, and NoSQL stores as needed, plus managed services and custom engines.
Notes ↗sql-trends · Beyond the Syntax: The Economics of Scale
The ETL to ELT flip: transform moves from before the load to after it
ETL runs Extract, Transform, then Load. ELT runs Extract, Load raw data into object storage, then Transform with SQL. Storage at about ten dollars per terabyte per month makes it feasible to keep every raw row and re-run transforms after a bug.
Notes ↗sql-trends · Beyond the Syntax: The Economics of Scale
Coupled versus decoupled storage and compute: pay for an idle 24/7 cluster, or pay per query
A coupled cluster ties 100 TB to always-on CPU and RAM and bills 24/7, even with weekly queries. A decoupled system stores raw data in object storage and rents compute only for the minutes a query runs.
Notes ↗sql-trends · Beyond the Syntax: The Economics of Scale
The ORM scale wall: an object-relational mapper wins in the startup phase and hits a wall at scale
An ORM maps Python objects to rows without SQL. In distributed systems, hidden ORM behavior can issue redundant queries and corrupt data without an obvious failure signal. Raw SQL knowledge supports a deliberate split between ORM code and handwritten queries.