Data Quality & Observability

Concept. You can load raw data quickly. You earn trust with two controls. You test invariants you can state up front, as declared assertions that block bad data before it ships. You observe properties you cannot state as fixed rules, by monitoring health signals and alerting on drift. A pipeline can report success while it produces wrong results.

Intuition. At Uber, an app update changed fare logging. A fare field disappeared for one in ten sessions in a few US cities. Nightly jobs completed successfully. Tests did not fail. A data scientist found the issue forty-five days later after millions of dollars. An assertion did not catch it because no one wrote a rule like "fares present in ninety-nine percent of Chicago sessions." Job success and data correctness diverged.

Raw data is dirty

Your quality checks are never finished, because you keep finding new ways the data breaks. That is why warehouses load raw and check downstream: a rule you add today still runs over everything you loaded last year.

Five cards showing the ways raw Spotify listen data arrives broken: a missing genre (NULL), a duplicate listen, a wrong type (text where a timestamp belongs), a late out-of-order event, and schema drift where an upstream rename changes a column. The clean field is slate, the corrupted value is red.

Figure 1. Raw pipelines fail in recurring ways. A value goes missing, a row duplicates, a field arrives with the wrong type, an event arrives late and out of order, or an upstream rename drifts the schema. Slate marks a clean field. Red marks a corrupted value that entered the warehouse.

Someone has to turn this into numbers a person can trust. That someone is the transform step, and it has two jobs: fix what you can name, and watch for what you cannot.

Test what you can predict

For every failure you can name in advance, write an assertion: a check that must hold on every run. In dbt (the tool that builds your tables from SQL) a test is a declared rule on a column, re-checked on every build. You attach the checks to the model in a config file:

# assertions on the top_genres model
columns:
  - name: genre
    tests: [not_null]        # every play must have a genre
  - name: listen_id
    tests: [unique]          # no play counted twice

This is the Module 1 lesson grown up: the same NULL that silently dropped a row in your first query now silently drops a whole product line at scale. This time, the test stops the build.

A four-stage dbt pipeline in slate (raw_listens, stg_listens, top_genres, dashboard). Under staging, unique(listen_id) passes in green and not_null(genre) fails in red; the failure raises a barrier that stops the build, so the top_genres mart and dashboard are greyed out and never receive the wrong number.

Figure 2. A failed test blocks the build. Each model runs as a `SELECT`. Each test asserts a property of the model output. `unique(listen_id)` passes (green). `not_null(genre)` fails (red) because thousands of podcast plays have no genre. The failure stops the pipeline, so the `top_genres` table never builds and the dashboard keeps yesterday's value. `NOT NULL` and `UNIQUE` constraints enforce the same properties at write time. dbt enforces them during transform.

Tests are cheap, they live next to the SQL, and they catch every failure you were smart enough to anticipate. Which is exactly their limit.

Observe what you cannot predict

The Uber fare bug had no test, because no one writes an assertion for every field in every city. Catching the failures you did not name is a different discipline: observability. Instead of checking a fixed rule, you learn what the data normally looks like and alert when it drifts. The trick is where you watch: an average over everything hides a broken slice, so you track each slice against its own baseline. The signals you watch:

  • Freshness (did today's data arrive on time?)

  • Volume (are the row counts in their usual range?)

  • Distribution (did the values shift, per slice?)

  • Schema (did a column change type or disappear?)

Two line charts. On the left, total plays across all cities stays inside its expected grey band, so the top-line looks healthy and no test fires. On the right, the same metric split by city shows Chicago's line falling out of its expected band in red, where a learned-baseline monitor fires an alert.

Figure 3. A global metric can stay inside its expected range while a slice breaks. Total plays remain within the expected band, so tests and dashboards stay quiet. A per-city split exposes Chicago falling outside its learned baseline (red), which triggers an alert. Uber built this approach in D3 and reduced detection time from forty-five days to two. Slate marks the metric. The grey band marks the expected range. Red marks the drifting slice.

You do not need to buy a product for this. A baseline is a rolling average with a normal range; open-source tools (Prophet is one) learn the band, and a nightly query compares today against it. The concept is the durable part, not any one product.

Run it, and gate on it

A scheduler ties the pieces together: it builds the tables, runs the tests, checks freshness, and does it every night in the right order, with retries. Prefect or Airflow are the common choices. The one rule that matters: when a test fails or the data is stale, the scheduler stops the pipeline and pages a human before the bad data spreads. Uber's platform does the same thing, suspending any pipeline whose inputs miss their quality bar.

One trap to name out loud. The scheduler tells you the job ran. The data checks tell you the data is right. These are not the same, and confusing them is how the Uber bug survived forty-five days: the job ran green every single night while the data quietly rotted.

Takeaway

Trust is a ladder, and each rung catches what the one below cannot.

Four slate steps rising left to right: constraints in the database at write time, dbt tests that block the failures you can name, observability that alerts on the ones you cannot, and an orchestrator (Prefect or Airflow) that runs it all and gates on the checks. Each higher rung catches what the lower one misses.

Figure 4. Constraints (`NOT NULL`, `UNIQUE`) enforce properties at write time. dbt tests block failures you can name before they ship. Observability alerts on failures you cannot name by detecting drift. An orchestrator runs the sequence on a schedule and gates downstream work on these checks. Slate marks a rung. Each higher rung covers cases that bypass lower rungs.

It is still just SQL and dependency graphs. It takes both: a correct query from Module 1 and correct data from this page. Get either one wrong and the number lies. Get both right and it is the only kind worth putting on a dashboard.