Small Data Is Underrated

There is no shortage of articles about Python. Most of them cover the happy path. This one is about what happened after the happy path ended.

When we finally measured, the numbers were worse than anyone guessed. Around 26% of the time spent on a typical change went into working around slowly changing dimensions rather than building anything.

What was going wrong

Most of the complexity was accidental. Over the years we'd layered three different approaches to slowly changing dimensions on top of each other, and each one made sense at the time.

The setup was ordinary: Python at the edges, Airflow doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

What we changed

Rather than a big-bang rewrite, we migrated one module at a time over about 5 weeks, keeping both paths alive behind a flag.

Here's a trimmed-down version of what we ended up with:

import polars as pl

daily = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event") == "signup")
    .group_by(pl.col("ts").dt.date().alias("day"))
    .agg(pl.len().alias("signups"))
    .sort("day")
    .collect()
)

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Measure before and after, even if it's just a rough number in a spreadsheet.
Keep the old path around until the new one has survived a real incident.
Delete dead code aggressively; it's the cheapest refactor there is.

Trade-offs

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Airflow for everything. That's a fair question with a boring answer: we tried, and it made slowly changing dimensions harder to reason about.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

Takeaways

If you take one thing away from this, let it be this: slowly changing dimensions is a design decision, not a cleanup task. Make it early and make it on purpose.