Why your dbt pipeline keeps breaking on API schema drift
Your dbt run was green yesterday. Today it's red because a marketing platform decided to rename a field, add a nested object, or quietly switch an integer to a string. Nobody told you. Nobody tells anyone. The vendor shipped a "minor update" and your revenue dashboards are now full of nulls.
This is schema drift, and if you're pulling data from ad platforms, ecommerce APIs, or any third-party source, it's not a rare edge case — it's the normal operating condition. The question isn't whether it happens. It's whether your pipeline survives it without paging someone at 2am.
The gap dbt was never meant to fill
dbt is a transformation tool. It assumes the data already landed in your warehouse in a consistent, typed, predictable shape. That assumption is doing a lot of work you probably haven't noticed — until an API changes and dbt starts throwing type errors on a column that used to be fine.
The real problem lives upstream, in extraction. If your ingestion layer just dumps whatever the API returns into a table and calls it done, you've built a pipeline with no shock absorber. Every upstream change propagates straight into your models, your tests, and eventually your CFO's dashboard. dbt didn't break. The thing feeding dbt broke, and dbt is just the messenger that happens to fail loudly.
Where schema drift actually comes from
It helps to name the specific failure modes instead of treating "the API changed" as one vague thing:
- Field renames — order_total becomes total_price with zero deprecation notice.
- Type changes — a field that was always an integer starts returning a string, often because the vendor added currency formatting.
- New nested structures — a flat field turns into an object with sub-fields, and your flattening logic silently drops data instead of erroring.
- Null behavior changes — a field that never used to be null suddenly is, because a new order type or channel got introduced on the platform side.
- Deprecated fields — the vendor keeps the field in the response but stops populating it, so your pipeline runs fine while the data quietly goes stale.
That last one is the worst. It doesn't fail. It just lies. Your dashboards keep updating with numbers that look plausible and are wrong.
Build a landing layer that expects to be lied to
The fix isn't a smarter dbt model. It's a raw landing layer designed with the assumption that the source will change without warning.
Concretely, that means:
- Land data as close to raw as possible — JSON blobs or variant columns — before you commit to a rigid schema. Let the warehouse hold the messy shape so a new field doesn't blow up the load step.
- Cast and validate types explicitly at the staging layer, not implicitly through whatever the loader guessed. If a field arrives as a string that should be numeric, you want a clear cast failure in staging, not a silent coercion three models downstream.
- Use schema contracts or explicit column lists in your staging models instead of select *. A wildcard select feels efficient until a new nested object shows up and doubles your row size or breaks a downstream join.
- Version your extraction logic against the API version, and treat API changelogs as a subscription, not a chore. Most platforms document breaking changes — the failure is usually that nobody on the data team reads them.
The goal is to move the point of failure as far upstream as possible, and to make it fail in an obvious, contained way — a staging model, not a finance report.
Tests catch drift, but only the drift you predicted
dbt tests are good at catching known failure modes: not-null checks, accepted values, referential integrity. They're bad at catching the thing you didn't think to test for, which is exactly what schema drift is by definition — a change you didn't anticipate.
That's where source freshness checks and schema tests on raw sources earn their keep. Instead of only testing your transformed models, test the shape of the raw data as it lands:
- Row count sanity checks against historical volume — a sudden 40% drop in rows from an API pull is a drift signal even if every test technically passes.
- Column presence checks on raw sources, so a dropped or renamed field trips an alert before it becomes a null cascade through five downstream models.
- Explicit "unknown column" alerts — some warehouses and ingestion tools can flag new fields that showed up in a payload but aren't mapped anywhere yet. Treat unmapped fields as information, not noise.
None of this eliminates drift. It just shrinks the time between "the API changed" and "someone on the data team knows about it" from days to minutes.
The real fix is architectural, not heroic
Teams that fight schema drift well don't have better dbt models. They have a clearer separation between extraction, landing, staging, and transformation — with validation happening at each boundary instead of hoping it all works out by the time you get to marts.
That separation also changes who gets paged and when. A break in raw ingestion should alert the person who owns the extraction script. A break in staging casting should alert whoever owns the contract with that source. A dbt model failing in the marts layer, three hops downstream from the actual problem, should be rare — because if it's common, it means your pipeline has no shock absorbers and every API hiccup becomes a full-stack incident.
Schema drift isn't a dbt problem. It's a pipeline design problem that shows up in dbt because dbt is usually the last thing to run and the first thing anyone looks at when numbers go wrong. Fix the layers before it, and dbt goes back to doing what it's actually good at: transforming data you can trust, instead of triaging data you can't.