Dealing With Late-Arriving Data

There is no shortage of articles about Snowflake. Most of them cover the happy path. This one is about what happened after the happy path ended.

The first symptom was subtle. Nothing failed outright, but late-arriving data kept drifting in the wrong direction, and every fix seemed to make the next one harder.

Where we started

Most of the complexity was accidental. Over the years we'd layered three different approaches to late-arriving data on top of each other, and each one made sense at the time.

The setup was ordinary: Snowflake at the edges, Airflow doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

Trying something different

We started by writing down the one rule we actually cared about, then deleted every bit of code that didn't serve it. It's not glamorous, but it works.

Here's a trimmed-down version of what we ended up with:

import polars as pl

daily = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event") == "signup")
    .group_by(pl.col("ts").dt.date().alias("day"))
    .agg(pl.len().alias("signups"))
    .sort("day")
    .collect()
)

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

• Measure before and after, even if it's just a rough number in a spreadsheet.
• Delete dead code aggressively; it's the cheapest refactor there is.
• Make the default the safe choice, not just the documented one.

What it cost us

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use pandas for everything. That's a fair question with a boring answer: we tried, and it made late-arriving data harder to reason about.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

Wrapping up

Six months in, nobody on the team wants to go back. That's about the best review a change like this can get.