Polars vs. pandas for Everyday Work

Every codebase has one area everyone is a little afraid to touch. On our team, it was everything related to slowly changing dimensions.

The first symptom was subtle. Nothing failed outright, but slowly changing dimensions kept drifting in the wrong direction, and every fix seemed to make the next one harder.

Some background

When we finally measured, the numbers were worse than anyone guessed. Around 42% of the time spent on a typical change went into working around slowly changing dimensions rather than building anything.

The setup was ordinary: pandas at the edges, DuckDB doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

How it works

We started by writing down the one rule we actually cared about, then deleted every bit of code that didn't serve it. It's not glamorous, but it works.

Stripped of the project-specific bits, it looks like this:

SELECT
  user_id,
  event_at,
  LAG(event_at) OVER (PARTITION BY user_id ORDER BY event_at) AS previous_event_at
FROM events
WHERE event_at >= CURRENT_DATE - INTERVAL '30 days';

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

• Write the migration guide before writing the migration.
• Delete dead code aggressively; it's the cheapest refactor there is.
• Keep the old path around until the new one has survived a real incident.

Caveats

We still haven't solved backfills. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

What's next

If you take one thing away from this, let it be this: slowly changing dimensions is a design decision, not a cleanup task. Make it early and make it on purpose.