Backfills Are a Product Feature

I've been working with Python for about 8 years now, and my opinions on backfills have changed more than I expected.

The setup was ordinary: Python at the edges, Spark doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

Some background

Most of the complexity was accidental. Over the years we'd layered three different approaches to backfills on top of each other, and each one made sense at the time.

When we finally measured, the numbers were worse than anyone guessed. Around 22% of the time spent on a typical change went into working around backfills rather than building anything.

How it works

We started by writing down the one rule we actually cared about, then deleted every bit of code that didn't serve it. It's not glamorous, but it works.

Stripped of the project-specific bits, it looks like this:

import polars as pl

daily = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event") == "signup")
    .group_by(pl.col("ts").dt.date().alias("day"))
    .agg(pl.len().alias("signups"))
    .sort("day")
    .collect()
)

It's tempting to generalize this into a library. Resist that urge until you've needed it in at least three places.

A few things that helped along the way:

• Keep the old path around until the new one has survived a real incident.
• Agree on what "done" means before the first pull request.
• Measure before and after, even if it's just a rough number in a spreadsheet.

What it cost us

The biggest cost was social, not technical. Changing how people think about backfills takes much longer than changing the code.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

What's next

If you take one thing away from this, let it be this: backfills is a design decision, not a cleanup task. Make it early and make it on purpose.