Incremental Models That Don't Lie to You

This post started as a note to my future self. Every few months I relearn the same lessons about incremental models, so this time I'm writing them down.

We had good intentions. We even had a document describing how incremental models should work. The trouble was that the document and the code had quietly gone their separate ways.

How we got here

Most of the complexity was accidental. Over the years we'd layered three different approaches to incremental models on top of each other, and each one made sense at the time.

The first symptom was subtle. Nothing failed outright, but incremental models kept drifting in the wrong direction, and every fix seemed to make the next one harder.

A simpler approach

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what Airflow already gives you out of the box.

Here's a trimmed-down version of what we ended up with:

import polars as pl

daily = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event") == "signup")
    .group_by(pl.col("ts").dt.date().alias("day"))
    .agg(pl.len().alias("signups"))
    .sort("day")
    .collect()
)

It's tempting to generalize this into a library. Resist that urge until you've needed it in at least three places.

A few things that helped along the way:

• Delete dead code aggressively; it's the cheapest refactor there is.
• Measure before and after, even if it's just a rough number in a spreadsheet.
• Pair on the first few changes so the knowledge doesn't live in one head.

What it cost us

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

We still haven't solved data quality checks. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

Wrapping up

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.