DuckDB Replaced Half of Our Data Stack

I've been working with DuckDB for about 9 years now, and my opinions on data quality checks have changed more than I expected.

The setup was ordinary: DuckDB at the edges, Polars doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

What was going wrong

Most of the complexity was accidental. Over the years we'd layered three different approaches to data quality checks on top of each other, and each one made sense at the time.

When we finally measured, the numbers were worse than anyone guessed. Around 38% of the time spent on a typical change went into working around data quality checks rather than building anything.

How it works

The key insight was to treat data quality checks as part of the interface rather than an implementation detail. Once we did that, most of the other decisions followed naturally.

Here's a trimmed-down version of what we ended up with:

import polars as pl

daily = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event") == "signup")
    .group_by(pl.col("ts").dt.date().alias("day"))
    .agg(pl.len().alias("signups"))
    .sort("day")
    .collect()
)

Nothing about this is clever, and that's the point. Anyone on the team can read it in a minute and know exactly what it does.

A few things that helped along the way:

Write the migration guide before writing the migration.
Pair on the first few changes so the knowledge doesn't live in one head.
Measure before and after, even if it's just a rough number in a spreadsheet.
Delete dead code aggressively; it's the cheapest refactor there is.

Where this breaks down

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Polars for everything. That's a fair question with a boring answer: we tried, and it made data quality checks harder to reason about.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

What's next

That's it for now. Next time I'll dig into schema drift, which turned out to be closely related.