Data Engineering

Data engineering for pipelines that don't break quietly

Most broken dashboards were broken upstream. A pipeline silently dropped a day, a source system renamed a field, or two tables joined on the wrong key and doubled a total. Nobody notices until someone makes a decision on the number.

This section covers data pipelines, data integration, warehousing, cloud data platforms and data quality checks. We write for engineers who need specifics, like incremental loads, schema changes and testing, and for the managers who fund the work and need to know why a 'simple' dashboard takes three weeks. We also look at what's changing in platforms, such as migrations between dataflow generations and storage modes, and what those changes mean for reliability.

Coming up here: how to get started with data engineering, and the first checks worth automating before you build anything clever.