Testing Pipelines Like Application Code

There is no shortage of articles about DuckDB. Most of them cover the happy path. This one is about what happened after the happy path ended.

We had good intentions. We even had a document describing how schema drift should work. The trouble was that the document and the code had quietly gone their separate ways.

The problem

When we finally measured, the numbers were worse than anyone guessed. Around 26% of the time spent on a typical change went into working around schema drift rather than building anything.

The setup was ordinary: DuckDB at the edges, Airflow doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

What we changed

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what DuckDB already gives you out of the box.

The core of it looks like this:

import polars as pl

daily = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event") == "signup")
    .group_by(pl.col("ts").dt.date().alias("day"))
    .agg(pl.len().alias("signups"))
    .sort("day")
    .collect()
)

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Keep the old path around until the new one has survived a real incident.
Write the migration guide before writing the migration.
Measure before and after, even if it's just a rough number in a spreadsheet.

Trade-offs

We still haven't solved data quality checks. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

The biggest cost was social, not technical. Changing how people think about schema drift takes much longer than changing the code.

Wrapping up

If you take one thing away from this, let it be this: schema drift is a design decision, not a cleanup task. Make it early and make it on purpose.