feat: build the physical stage from the dbt project, not the manifest - #62
Merged
Merged
Conversation
The stage named Physical has never actually seen a warehouse. dbt copies manifest.json's column types straight out of the schema YAML, so what the canvas showed as physical types was the user's own documentation reflected back — and because "exists" meant "present in the compiled manifest", a project that had not run dbt rendered every model as a grey ghost, saying nothing here exists when it meant you have not run dbt lately. Existence now comes from the dbt project itself: a source file under the model, seed or snapshot paths, or a schema YAML, manifest node or catalog relation naming it, minus anything dbt has disabled. target/catalog.json is read when present — the one artifact that has looked at the warehouse — unioned with the declared columns rather than replacing them, because dbt docs generate runs far less often than dbt run and a catalog-wins rule would hide a column added an hour ago. Each model now reports where its shape came from, so a type on the canvas is never an unattributed guess. Type comparison is rebuilt on the spellings adapters actually emit, and a type declared on only one side reports as undeclared rather than inflating the mismatch count. Sync plans carry a stage-absolute resolvedDataType, because the existing pair is named for the comparison direction and an assistant following the old harness table could write the empty side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ywafwDpxdWWBt26Q1keiP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.sql/.py/.csvsits under the model, seed or snapshot paths, or a schema YAML, manifest node or catalog relation names it — minus anything dbt has disabled. Previously "exists" meant "in the compiled manifest", so a project with nomanifest.jsonrendered every model as a grey ghost. A model designed but not yet built now ghosts (with a reason) instead of vanishing silently.target/catalog.jsonis read when present — the only dbt artifact that has actually seen the warehouse. Real column types and undocumented columns, unioned with the declared list rather than replacing it, sincedbt docs generateruns far less often thandbt run.data_type:→ manifest → blank, never invented.NUMBER/TEXT, Postgrescharacter varying(255), BigQueryARRAY<…>), plus a newundeclaredstatus so an absentdata_type:stops inflating the mismatch count.resolvedDataType— stage-absolute, so an assistant can no longer write the empty side of anundeclaredcolumn.Test plan
npm run compile,npm run buildnpm test— 62 files / 1510 tests (was 1403)tsc --noEmit,npm run build,node test-smoke.mjsvsce ls --no-dependencies→ 12 files (limit 60)absentvsdisabledghosts, and the no-artifacts noticeHARNESS_VERSION16 → 17 bump is wanted in this release (it prompts every user who has installed harness files)🤖 Generated with Claude Code
https://claude.ai/code/session_014ywafwDpxdWWBt26Q1keiP