Small Thread

data engineering

Small Thread

Hi, I'm Miles. A slow blog about data engineering and the craft of shipping software. I write about late-arriving data, schema drift, slowly changing dimensions and whatever else I'm learning this month.

About meSubscribe via RSS

Latest writing

  • · 3 min read

    Partitioning Strategies I Wish I'd Known Earlier

    I've been working with dbt for about 3 years now, and my opinions on partitioning have changed more than I expected.

    #analytics#pipelines
  • · 3 min read

    Backfills Are a Product Feature

    I've been working with Python for about 8 years now, and my opinions on backfills have changed more than I expected.

    #analytics#duckdb#warehouse
  • · 3 min read

    Data Contracts Without the Hype

    For a long time I treated data contracts as something to worry about later. Then a small change to our Airflow setup turned into a week of debugging, and "later" arrived all at once.

    #python#dbt#data-quality
  • · 3 min read

    Testing Pipelines Like Application Code

    There is no shortage of articles about DuckDB. Most of them cover the happy path. This one is about what happened after the happy path ended.

    #sql#python#duckdb
  • · 3 min read

    Small Data Is Underrated

    There is no shortage of articles about Python. Most of them cover the happy path. This one is about what happened after the happy path ended.

    #warehouse#pipelines#sql
  • · 3 min read

    Polars vs. pandas for Everyday Work

    Every codebase has one area everyone is a little afraid to touch. On our team, it was everything related to slowly changing dimensions.

    #sql#pipelines
  • · 3 min read

    The Metric Everyone Defined Differently

    There is no shortage of articles about DuckDB. Most of them cover the happy path. This one is about what happened after the happy path ended.

    #data-quality#analytics
  • · 3 min read

    DuckDB Replaced Half of Our Data Stack

    I've been working with DuckDB for about 9 years now, and my opinions on data quality checks have changed more than I expected.

    #duckdb#sql
  • · 3 min read

    Incremental Models That Don't Lie to You

    This post started as a note to my future self. Every few months I relearn the same lessons about incremental models, so this time I'm writing them down.

    #dbt#sql#pipelines
  • · 3 min read

    Dealing With Late-Arriving Data

    There is no shortage of articles about Snowflake. Most of them cover the happy path. This one is about what happened after the happy path ended.

    #analytics#pipelines