Cutting a nightly scan by 95% and ending the timeouts

A multi-brand data warehouse — 5 brands, one dbt project, 230+ models — kept failing on a nightly timeout. Two months of raising the timeout hadn't fixed it, because the real cause wasn't slow models. It was every model scanning the entire history, every single night.

The problem

The nightly dbt run kept failing on a 3,600-second timeout — but a different model every night, green again after a retry, so for two months the fix was "raise the timeout." The real cause was structural: none of the source tables were partitioned, so every model rescanned the entire history every night, and a handful of heavy models fought each other for BigQuery slots. Other jobs running in the same project made the picture look random.

What we did

• Measured instead of guessing. Broke down the nightly scan by model (job stats by node_id) and found that 27 brand-specific models — not the "heavy" core models everyone suspected — accounted for 52.7% of all bytes scanned.
• Added incremental guards. A _TABLE_SUFFIX window on every source branch, plus a partition-column clamp around the model body, with correct behavior on --full-refresh. Rolled out identically across 4 brands.
• Rebuilt the run itself. Two sequential phases instead of one free-for-all (the heavy block isolated, single-threaded), interactive priority instead of batch (batch was hiding slot starvation as a "stuck job"), and per-model recovery with kill+retry instead of failing the whole run.
• Guarded against silent regressions. A byte-for-byte diff of the compiled SQL before and after every change — it caught two edits that landed inside a comment and never actually ran.

The result

Nightly scan on the brand models dropped from 584.6 GiB to 27.0 GiB — a 95.4% reduction — and the full nightly run went from 1.63 TiB to about 1.09 TiB. Nightly timeouts stopped happening entirely, and BigQuery cost fell in proportion to the scan. The output didn't change: equivalence was proven by comparing values, not just row counts.