Working with dbt?

dbt project refusing to compile or tests failing silently? We unstick it.

A dbt project grows from twenty models to five hundred without anyone noticing, and then one morning a scheduler run fails on a model nobody owns, a test is marked warn when it should be error, or state:modified selects the wrong set of models and ships a half-applied change. We work through the manifest, the macro layer and the warehouse logs until the project is predictable again and the daily run is boring.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

dbt-run.log

still failing

What it is

dbt at a glance

dbt is the analytics engineering standard for transforming data inside a warehouse. The open source dbt-core runs models written as SELECT statements, resolves their dependency graph, materializes them as tables or views, and ships tests and documentation alongside. dbt Cloud wraps it in a hosted scheduler, IDE and lineage browser.

By 2026 dbt is the default tool for the T in ELT across Snowflake, BigQuery, Databricks, Redshift and Postgres. Teams organize models into staging, intermediate and marts layers, pin sources with freshness checks, and use tests to guard against the kinds of silent data bugs that used to go undetected for weeks. Mesh, semantic layer and unit tests are now first-class features in the toolchain.

Type
Analytics engineering framework for data warehouses
First released
2016 as Fishtown Analytics' internal tool
License
dbt-core is Apache 2.0, dbt Cloud is proprietary
Maintained by
dbt Labs
Typically used for
ELT transformation inside Snowflake, BigQuery, Databricks, Redshift and Postgres, data modeling, testing, documentation

The honest version

dbt pros and cons

No technology is all upside. This is where dbt earns its place and where it tends to cause trouble.

Where dbt is strong

  • SQL as the transformation languageAnalysts and data engineers write models in SQL that the warehouse already speaks, with Jinja for the parts SQL cannot express. The learning curve is far smaller than a custom ETL framework.
  • A real dependency graphref() calls build a DAG that dbt walks in order, which means transformations run in the right sequence automatically and parallelism comes for free up to the warehouse's throughput.
  • Tests are first-classunique, not_null, relationships and accepted_values tests catch most of the silent data bugs that used to go to production. Custom tests cover the rest with the same ergonomics.
  • Documentation and lineage come as a byproductdbt docs generate a browsable lineage graph from the same metadata the project uses to run, so documentation stays close to the code that produced it.
  • Package ecosystem is strongdbt_utils, dbt_expectations, codegen, and the Fivetran and Snowplow packages cover huge amounts of common ground. Starting a project means importing patterns rather than inventing them.
  • Portable across warehousesThe same project model works on Snowflake, BigQuery, Databricks, Redshift and Postgres with adapter-specific configuration. Migrations between warehouses are still work, but far less than rewriting SQL from scratch.

Where dbt bites

  • Compile time grows with the projectA thousand-model project can take minutes to parse and manifest, which makes local iteration slow. Deferred manifests and state selection help, but the dev loop still shortens with discipline rather than magic.
  • Jinja makes SQL harder to readMacros are powerful and also one of the main sources of hard-to-debug problems. A macro that produces different SQL in dev and prod is easy to write and painful to trace.
  • Test configuration is fiddlyA test set to warn instead of error, a severity threshold that masks a real failure, or a store_failures table nobody checks can all mean the daily run looks green while bad data is landing.
  • Incremental models have sharp edgesBackfills, unique_key enforcement, merge versus insert_overwrite behavior and late-arriving data all require care. The default append strategy lies to you in subtle ways.
  • State and defer are confusingstate:modified and defer let CI run only the changed models against a production manifest, which is powerful and also a common source of bugs when the manifest drifts from reality.
  • Warehouse cost is on your shouldersdbt is cheap; the warehouse running the SQL is not. A full refresh of a wide table, or an incremental model that forgot to filter the source, can add hundreds of dollars to a single run.

Sound familiar?

Common dbt problems we fix

The dbt problems that bring people to us, and what you have once each one is fixed.

Failure in test unique_users_id: Got 3 records

A unique test is failing in production

A source that started shipping duplicates, a join that was always slightly wrong, or a window that moved and started picking up data twice. The test caught it; the fix is to trace the duplication back to its origin.

Duplicates found and removed at source

Freshness check failed: raw.events is 8 hours stale

Source freshness alerts are firing

An ingestion job that paused, a connector that moved to a new table name, or a source that changed its schedule. The dbt side is working; the pipeline feeding it is behind.

Sources fresh on expected cadence

Node depends on a node named 'stg_events' which was not found

A ref() points at a model that moved

A model that was renamed or deleted, or a feature branch that is missing the dependency the main project expects. Shows up on CI before production, if CI is wired up.

Dependencies resolve on every branch

Manifest version mismatch

dbt-core upgrade breaks CI

A deploy that bumped dbt-core without regenerating the manifest, so state:modified, defer and docs all go sideways. Easy to miss because local runs still work.

Core version and manifest in step

Incremental model: unique_key not defined

Incremental model silently duplicates rows

An incremental configured without a unique_key, which falls back to append and starts producing duplicate records on every run. Nobody notices until downstream counts drift.

Incremental strategy explicit

state:modified selector returned 0 nodes

CI run skips the real changes

A state comparison against a stale manifest, or a path selector that missed the actual files changed in the pull request. The slim CI run goes green on a broken project.

State comparison trustworthy

Snowflake query timeout

A model exceeds the warehouse statement timeout

A full refresh of a very large model on a warehouse size that cannot finish it inside the configured timeout. Either the warehouse size is wrong or the model should be incremental.

Model shape matches warehouse size

Permission denied on resource analytics.dbt_prod

Grants drift between environments

A new model that writes to a schema whose grants were set up by hand, so dev works and prod does not. Grants in configuration rather than in the dashboard fix the class of bug.

Grants managed in code

What we do

dbt help, from one failing test to a healthy project

Call us for one failing model, or hand us the whole analytics project.

Rescue
A scheduler run that keeps failing, a test suite nobody trusts, or a manifest that is out of sync. We stabilize the daily run first, then clean the structure.
Project structure
Staging, intermediate and mart layering, naming conventions, shared macros and package choices worked out against your warehouse and team shape.
Test and freshness strategy
Tests at the right severity, source freshness that fires before downstream breaks, and a review of what is masked by warn thresholds.
Incremental and performance work
Incremental models with explicit unique_keys and merge strategies, warehouse sizing matched to run times, and a cost review of the heaviest models.
CI and state
A CI pipeline that uses defer and state:modified correctly, manifest storage that keeps the slim build trustworthy, and a release path from dev to prod that does not surprise anyone.
Managed Tech
Standing care for the project: run health, cost trends, package and core upgrades, documentation drift, and a line to call when a mart stops matching finance's numbers.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running dbt alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

dbt questions

Something else? Call+1 737 727 7327

How do we structure a dbt project that will grow past a hundred models?

A staging layer that cleans and renames source data without joining, an intermediate layer where the heavy transformations live, and a marts layer that is what the business actually reads. Add generic tests early, pin sources with freshness, and keep the macro layer small enough that one person can hold the whole thing in their head.

Why do our dbt tests go green while the data is wrong?

Almost always one of three things: the test severity is set to warn instead of error, the store_failures output is not inspected, or the test is written against the output rather than against the invariant that should hold. We audit the test configuration as part of any project review.

How do we keep dbt warehouse costs under control?

The big wins are incremental models with correct unique_keys, warehouse sizes matched to the model shape rather than the biggest query in the project, and a review of models that run every hour when they could run daily. dbt's query profile plus the warehouse's query history give you everything you need to find the heavy ones.

dbt Core or dbt Cloud?

Core is free and runs anywhere; Cloud adds a scheduler, IDE, lineage browser and CI integration. For small teams the Core plus a scheduler like Airflow or GitHub Actions path is cost effective. For larger teams the Cloud features often pay for themselves in time saved. We have shipped both.

How do we handle incremental models that keep drifting?

Set unique_key explicitly, pick merge or insert_overwrite rather than letting it default to append, and add a full-refresh schedule that catches late-arriving data. A reconciliation test that compares the incremental to a full refresh on a cadence surfaces drift before downstream users do.

How much does dbt help cost?

The first review is free. Share the project scale and the error that is biting, and we come back with a price before any work begins.

Free estimate

dbt giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.