fix: stop the templates and samples teaching the wrong key - #66
Merged
Merged
Conversation
Raised by Joakim Dalby on a LinkedIn post showing the demo GIF: the
customer dimension keyed on customer_id while flagging customer_name as
the natural key. Names are not unique, so that is a claim the data
cannot honour — and the defect was not confined to the sample.
FALLBACK_TEMPLATES, what every user without custom templates inherits,
started each dimension with `{name}_id`, described it as a "Surrogate
key" and made it the primary key. That conflates two different things:
{name}_key — the surrogate. Meaningless, unique per ROW, the column
facts point at.
{name}_id — the business key, as the source system spells it. Unique
per ENTITY, which is not the same: under SCD2 one entity
owns a row per version, so it repeats by design and
cannot be the primary key.
The SCD Type 2 template therefore shipped a dimension whose declared
primary key is provably not unique — and ERD Studio derives cardinality
from uniqueness, so it was not only cosmetic.
The AI harness in SCHEMA_CONTENT already documents this correctly
(`email` as the natural key, `full_name` as a tracked attribute), so
HARNESS_VERSION is deliberately not bumped: the guidance was right and
the defaults contradicted it.
- Templates: dimensions and scd2 split `{name}_key` / `{name}_id`; scd2
marks its tracked attributes scdType 2; the Type 1 template drops the
Type 2 effective dating and scd2 drops its duplicate set; bridge flags
its two sides isForeignKey rather than only claiming so in prose.
- Sample models: every dimension gets a `_key` surrogate and a real
business key (`customer_id`, `project_code`, `location_code`,
`task_id`), names become ordinary tracked attributes, and facts join
to dimension surrogates. Renames carried through the domain
relationships, the dbt schema yml, manifest.json and catalog.json —
the catalog keeps its deliberate UPPERCASE spelling for dim_task.
- dim_date is unchanged: a YYYYMMDD key is the conventional exception,
now documented as one, with calendar_date as its business key.
- showcase gains dim_customer and dim_date, which sat in the library
referenced by nothing, so the dimension that best shows the split was
never on the canvas.
- media/demo.gif regenerated from the corrected models. Every frame used
to show the NK badge on customer_name.
New test/unit/dimensionalModelling.test.ts asserts the invariants (one
PK per dimension, PK and NK are different columns, no label-shaped
column is ever the natural key, fact FKs end `_key`) so this cannot
regress. Verified to fail against the previous fixtures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137crR8usmNLiUke7NeYpBY
The canvas-only GIF lost the thing the original sold — that this runs in your editor, next to the files it reads. The editor area is still the real dist/webview.js rendering real payloads; the window around it (title bar, activity bar, explorer, tabs, status bar) is drawn in CSS. It is drawn rather than screen-captured, so it is held to being true: the file tree lists exactly what test/fixtures/dbt-project contains, the tab and breadcrumb name the file actually open, the activity bar carries the real ERD Studio icon from media/icon-sidebar.svg, and the language indicator says JSON because the open file is a domain JSON. Nothing on screen names a file the repository does not have or claims a feature the extension does not ship. The README alt text and the CHANGELOG say the chrome is rendered, so nobody later mistakes it for a screenshot. 467 KB, 1400x929, four frames. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137crR8usmNLiUke7NeYpBY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prompted by Joakim Dalby's comment on a LinkedIn post showing the demo GIF:
They were being polite. It was the name.
What was actually wrong
dim_locationdid the same withsite_name. But the sample was the symptom, not the disease —FALLBACK_TEMPLATESinsrc/services/templateService.tshas it too, and that's what every user without custom templates inherits.Two things are being conflated:
{name}_key{name}_idThose are not the same, and SCD2 is where it bites: one entity owns a row per version, so the business key repeats by design. The
scd2template therefore shipped a dimension whose declared primary key is provably not unique — and ERD Studio derives cardinality from uniqueness tests, so that isn't only cosmetic.The AI harness already had this right.
SCHEMA_CONTENTdocumentsemailas the natural key andfull_nameas a tracked attribute. SoHARNESS_VERSIONis deliberately not bumped: the guidance was correct, the defaults contradicted it.What changed
Templates (
FALLBACK_TEMPLATES+ the fixture copies)dimension/scd2split{name}_keyand{name}_id, the latter flaggedisNaturalKeyscd2marks its tracked attributesscdType: 2, so it finally demonstrates the feature it's named afterdimension(Type 1) drops the Type 2 effective dating it was shipping;scd2drops its duplicate set (it carried bothvalid_from/valid_toandscd_valid_from/scd_valid_to)bridgeflags its two sidesisForeignKeyinstead of only claiming so in the description textSample models — every dimension gets a
_keysurrogate and a real business key (customer_id,project_code,location_code,task_id); names become ordinary tracked attributes; facts join to dimension surrogates. Carried through the domain relationships, the dbt schema yml,manifest.jsonandcatalog.json— the catalog keeps its deliberate UPPERCASE spelling fordim_task, which is what proves the declared spelling wins for display.dim_dateis untouched. AYYYYMMDDkey is the one conventional exception — exactly the concession Joakim made — and it's now documented as such, withcalendar_dateas its business key.showcasegainsdim_customeranddim_date. Both sat in the model library referenced by nothing, so the dimension that best demonstrates the split was never on the canvas.The demo GIF
Every frame of
media/demo.gif— the marketplace README hero — showeddim_customerwith an NK badge oncustomer_name. Regenerated:The canvas is real; the window around it is drawn. The editor area in each frame is the actual
dist/webview.jsbundle, driven by payloads a realSemanticEditorProviderproduced over the real fixture project — the logicaldomainLoaded, the physicalstageData, and thediscrepancyReportfrom an actualtoggleDiscrepancy. Every caption is asserted against the live DOM while capturing, so a mismatch aborts the run.The VS Code window (title bar, activity bar, explorer, tabs, status bar) is CSS, not a screen capture — there's no display or VS Code in the environment this was built in. Because it's drawn, I held it to being true rather than merely plausible:
test/fixtures/dbt-projectcontainsmedia/icon-sidebar.svgJSON, because a domain file is JSONNothing on screen names a file the repo doesn't have or implies a feature that doesn't ship. The README alt text opens with "ERD Studio open in VS Code" and the CHANGELOG states the chrome is rendered, so this can't later be mistaken for a screenshot. If you'd rather have a genuine capture, the fixture state is committed — recapturing in the dev host is a couple of minutes.
One honest detail in frame 4: the overlay reports
customer_id Only in Logical. That's correct — the new business key is in the design and not yet in the warehouse, which is precisely what the discrepancy view is for.467 KB, 1400×929, four frames.
Tests
1560pass (1527before),33new intest/unit/dimensionalModelling.test.ts, asserting the rules rather than the current strings:*_name,*_label,*_title) as the natural keyscd2tracks history on attributes and on neither key, and carries the columns that make versions addressablescd2ships only one set_key; a fact keys itself on a degenerate business identifierdim_dateis the documented exceptionVerified they fail against the previous fixtures — reverting
dim_customer.ymlalone trips two of them with the exact complaint Joakim made.npm run compile,npm run build,npm testgreen;vsce ls --no-dependenciesstill 12 files.Worth pushing back on
{name}_keyis a convention, not a law. Plenty of dbt practice skips integer surrogates for a hash (dbt_utils.generate_surrogate_key) or uses the business key directly. I picked the explicit split because it's the one a template can teach, and a template that stays silent teaches the conflation by default — but it is a choice. The part with no other side is that a name is not a key, and that's enforced generically rather than by naming specific columns.{name}_idas both PK and NK — a degenerate dimension — rather than gaining a surrogate that buys nothing. If you'd rather facts carried{name}_keytoo, it's a small change to the templates and the generic tests still hold.🤖 Generated with Claude Code
https://claude.ai/code/session_0137crR8usmNLiUke7NeYpBY