Skip to content

fix: stop the templates and samples teaching the wrong key - #66

Merged
liam-machine merged 2 commits into
mainfrom
claude/investigate-repo-issue-tr02yn
Sep 20, 2026
Merged

liam-machine merged 2 commits into
mainfrom
claude/investigate-repo-issue-tr02yn

Conversation

@liam-machine

@liam-machine liam-machine commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Prompted by Joakim Dalby's comment on a LinkedIn post showing the demo GIF:

I wish your customer dimension has a dimension key called Customer_key, because I think the business key is Customer_Id, and hopefully not the name of customer since name of persons is not unique. Date_key can be a date.

They were being polite. It was the name.

What was actually wrong

# dim_customer.yml, before
- name: customer_id
  description: Surrogate key for customer    # a surrogate, named like a business key
  isPrimaryKey: true
- name: customer_name
  isNaturalKey: true                         # the name IS marked as the business key

dim_location did the same with site_name. But the sample was the symptom, not the disease — FALLBACK_TEMPLATES in src/services/templateService.ts has it too, and that's what every user without custom templates inherits.

Two things are being conflated:

{name}_key surrogate meaningless, unique per row, the column facts point at
{name}_id business key as the source system spells it, unique per entity

Those are not the same, and SCD2 is where it bites: one entity owns a row per version, so the business key repeats by design. The scd2 template therefore shipped a dimension whose declared primary key is provably not unique — and ERD Studio derives cardinality from uniqueness tests, so that isn't only cosmetic.

The AI harness already had this right. SCHEMA_CONTENT documents email as the natural key and full_name as a tracked attribute. So HARNESS_VERSION is deliberately not bumped: the guidance was correct, the defaults contradicted it.

What changed

Templates (FALLBACK_TEMPLATES + the fixture copies)

  • dimension / scd2 split {name}_key and {name}_id, the latter flagged isNaturalKey
  • scd2 marks its tracked attributes scdType: 2, so it finally demonstrates the feature it's named after
  • dimension (Type 1) drops the Type 2 effective dating it was shipping; scd2 drops its duplicate set (it carried both valid_from/valid_to and scd_valid_from/scd_valid_to)
  • bridge flags its two sides isForeignKey instead of only claiming so in the description text

Sample models — every dimension gets a _key surrogate and a real business key (customer_id, project_code, location_code, task_id); names become ordinary tracked attributes; facts join to dimension surrogates. Carried through the domain relationships, the dbt schema yml, manifest.json and catalog.json — the catalog keeps its deliberate UPPERCASE spelling for dim_task, which is what proves the declared spelling wins for display.

dim_date is untouched. A YYYYMMDD key is the one conventional exception — exactly the concession Joakim made — and it's now documented as such, with calendar_date as its business key.

showcase gains dim_customer and dim_date. Both sat in the model library referenced by nothing, so the dimension that best demonstrates the split was never on the canvas.

The demo GIF

Every frame of media/demo.gif — the marketplace README hero — showed dim_customer with an NK badge on customer_name. Regenerated:

demo

The canvas is real; the window around it is drawn. The editor area in each frame is the actual dist/webview.js bundle, driven by payloads a real SemanticEditorProvider produced over the real fixture project — the logical domainLoaded, the physical stageData, and the discrepancyReport from an actual toggleDiscrepancy. Every caption is asserted against the live DOM while capturing, so a mismatch aborts the run.

The VS Code window (title bar, activity bar, explorer, tabs, status bar) is CSS, not a screen capture — there's no display or VS Code in the environment this was built in. Because it's drawn, I held it to being true rather than merely plausible:

  • the file tree lists exactly what test/fixtures/dbt-project contains
  • the tab and breadcrumb name the file that's actually open
  • the activity bar carries the real ERD Studio icon from media/icon-sidebar.svg
  • the language indicator reads JSON, because a domain file is JSON

Nothing on screen names a file the repo doesn't have or implies a feature that doesn't ship. The README alt text opens with "ERD Studio open in VS Code" and the CHANGELOG states the chrome is rendered, so this can't later be mistaken for a screenshot. If you'd rather have a genuine capture, the fixture state is committed — recapturing in the dev host is a couple of minutes.

One honest detail in frame 4: the overlay reports customer_id Only in Logical. That's correct — the new business key is in the design and not yet in the warehouse, which is precisely what the discrepancy view is for.

467 KB, 1400×929, four frames.

Tests

1560 pass (1527 before), 33 new in test/unit/dimensionalModelling.test.ts, asserting the rules rather than the current strings:

  • a dimension has exactly one primary key, and its PK and NK are different columns
  • no template or sample ever flags a label-shaped column (*_name, *_label, *_title) as the natural key
  • scd2 tracks history on attributes and on neither key, and carries the columns that make versions addressable
  • the Type 1 template ships no Type 2 dating, and scd2 ships only one set
  • fact foreign keys end _key; a fact keys itself on a degenerate business identifier
  • dim_date is the documented exception

Verified they fail against the previous fixtures — reverting dim_customer.yml alone trips two of them with the exact complaint Joakim made.

npm run compile, npm run build, npm test green; vsce ls --no-dependencies still 12 files.

Worth pushing back on

  • {name}_key is a convention, not a law. Plenty of dbt practice skips integer surrogates for a hash (dbt_utils.generate_surrogate_key) or uses the business key directly. I picked the explicit split because it's the one a template can teach, and a template that stays silent teaches the conflation by default — but it is a choice. The part with no other side is that a name is not a key, and that's enforced generically rather than by naming specific columns.
  • The fact PK. Facts keep {name}_id as both PK and NK — a degenerate dimension — rather than gaining a surrogate that buys nothing. If you'd rather facts carried {name}_key too, it's a small change to the templates and the generic tests still hold.

🤖 Generated with Claude Code

https://claude.ai/code/session_0137crR8usmNLiUke7NeYpBY

Raised by Joakim Dalby on a LinkedIn post showing the demo GIF: the
customer dimension keyed on customer_id while flagging customer_name as
the natural key. Names are not unique, so that is a claim the data
cannot honour — and the defect was not confined to the sample.

FALLBACK_TEMPLATES, what every user without custom templates inherits,
started each dimension with `{name}_id`, described it as a "Surrogate
key" and made it the primary key. That conflates two different things:

  {name}_key — the surrogate. Meaningless, unique per ROW, the column
               facts point at.
  {name}_id  — the business key, as the source system spells it. Unique
               per ENTITY, which is not the same: under SCD2 one entity
               owns a row per version, so it repeats by design and
               cannot be the primary key.

The SCD Type 2 template therefore shipped a dimension whose declared
primary key is provably not unique — and ERD Studio derives cardinality
from uniqueness, so it was not only cosmetic.

The AI harness in SCHEMA_CONTENT already documents this correctly
(`email` as the natural key, `full_name` as a tracked attribute), so
HARNESS_VERSION is deliberately not bumped: the guidance was right and
the defaults contradicted it.

- Templates: dimensions and scd2 split `{name}_key` / `{name}_id`; scd2
  marks its tracked attributes scdType 2; the Type 1 template drops the
  Type 2 effective dating and scd2 drops its duplicate set; bridge flags
  its two sides isForeignKey rather than only claiming so in prose.
- Sample models: every dimension gets a `_key` surrogate and a real
  business key (`customer_id`, `project_code`, `location_code`,
  `task_id`), names become ordinary tracked attributes, and facts join
  to dimension surrogates. Renames carried through the domain
  relationships, the dbt schema yml, manifest.json and catalog.json —
  the catalog keeps its deliberate UPPERCASE spelling for dim_task.
- dim_date is unchanged: a YYYYMMDD key is the conventional exception,
  now documented as one, with calendar_date as its business key.
- showcase gains dim_customer and dim_date, which sat in the library
  referenced by nothing, so the dimension that best shows the split was
  never on the canvas.
- media/demo.gif regenerated from the corrected models. Every frame used
  to show the NK badge on customer_name.

New test/unit/dimensionalModelling.test.ts asserts the invariants (one
PK per dimension, PK and NK are different columns, no label-shaped
column is ever the natural key, fact FKs end `_key`) so this cannot
regress. Verified to fail against the previous fixtures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137crR8usmNLiUke7NeYpBY
The canvas-only GIF lost the thing the original sold — that this runs in
your editor, next to the files it reads. The editor area is still the
real dist/webview.js rendering real payloads; the window around it
(title bar, activity bar, explorer, tabs, status bar) is drawn in CSS.

It is drawn rather than screen-captured, so it is held to being true:
the file tree lists exactly what test/fixtures/dbt-project contains, the
tab and breadcrumb name the file actually open, the activity bar carries
the real ERD Studio icon from media/icon-sidebar.svg, and the language
indicator says JSON because the open file is a domain JSON. Nothing on
screen names a file the repository does not have or claims a feature the
extension does not ship. The README alt text and the CHANGELOG say the
chrome is rendered, so nobody later mistakes it for a screenshot.

467 KB, 1400x929, four frames.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137crR8usmNLiUke7NeYpBY
@liam-machine
liam-machine merged commit f835c34 into main Sep 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants