Skip to content

Warn when relationship to_columns does not cover a declared key - #330

Open
kayemkim wants to merge 1 commit into
apache:mainfrom
kayemkim:validate-to-columns-key-coverage
Open

Warn when relationship to_columns does not cover a declared key#330
kayemkim wants to merge 1 commit into
apache:mainfrom
kayemkim:validate-to-columns-key-coverage

Conversation

@kayemkim

Copy link
Copy Markdown

Summary

The spec defines to_columns as "Primary/unique key columns in the 'to' dataset" (core-spec/spec.md, osi-schema.json), but the validator only checks that the relationship's datasets exist. A relationship that joins to non-key columns silently breaks many-to-one semantics downstream — joins fan out, and consumers that trust the declared cardinality produce wrong results. #301 shows this class of document being produced in practice and rejected by downstream consumers; a validator-side check catches it at the document level regardless of which tool produced it.

validate_references now checks that to_columns covers the to dataset's primary_key or one of its unique_keys. Two deliberate softenings, both drawn from how the repo already treats this rule:

  • Coverage rather than exact equality: a to_columns that is a superset of a key still guarantees the join cardinality (tenant-sharded joins are a common shape), and the Databricks converter's _covers_unique_key already applies exactly these semantics.
  • Warning rather than error: declared keys can be a partial recovery of the dataset's real keys (the Databricks importer, for instance, records unique_keys from a single join's rely hint), so a non-covering to_columns is suspicious but not provably wrong. The message goes through the validator's existing warning channel and does not fail validation. Happy to tighten this to an error if you'd rather trust key declarations as exhaustive.

Datasets that declare no keys are skipped — both primary_key and unique_keys are optional, so there is nothing to check against. The check also guards against shapes that already fail schema validation (null unique_keys, non-list to_columns, flat unique_keys) so the semantic pass can't crash or mislead on an invalid document.

Ran against every semantic-model YAML in the repo (examples plus converter fixtures): no regressions, all still pass. A relationship pointing at non-key columns now reports:

[Reference] Warning: Relationship 'orders_to_customers' in model 'm': to_columns ['region'] does not cover the primary key or a unique key of dataset 'customers'

A couple of adjacent gaps I noticed but left out of scope, flagging in case they're worth issues: no CI workflow currently runs validation/tests (this suite included), and the orionbelt converter's mirror validator (which documents itself as mirroring validate.py) doesn't have this rule — I can follow up on either. #307 (pending) adds from/to column count validation to the same function and introduces the same test file path; the checks are complementary and I'm happy to rebase if it lands first.

Related Issues

Related to #301 (validator-side guard for the same class of invalid document; does not fix the converter itself).

Checklist

Specification

  • Spec changes are included in core-spec/ and follow the existing structure
  • Spec changes have been discussed on the mailing list or in a linked issue
  • Breaking changes to the spec are clearly called out in the summary

Ontology

  • Ontology changes in ontology/ are consistent with spec changes
  • New or modified terms are defined and documented

Converters

  • Converter logic in converters/ is updated to reflect spec or ontology changes
  • New converters include tests under the converter's test directory

Validation

  • Validation rules in validation/ are updated if the spec changed
  • New validation cases are covered by tests

Documentation

  • docs/ is updated to reflect any user-facing changes
  • New features or behaviors are documented with examples where appropriate
  • CONTRIBUTING.md is updated if the contribution process changed

Examples

  • examples/ are added or updated for any new spec constructs or converter support

Tests

  • All existing tests pass (pytest / CI green)
  • New functionality is covered by tests

Compliance

  • ASF license headers are present on all new source files
  • No third-party dependencies are added without PMC/IPMC approval

The spec defines to_columns as "Primary/unique key columns in the 'to'
dataset", but the validator only checked that relationship datasets
exist. A relationship joining to non-key columns breaks many-to-one
semantics downstream (see apache#301 for a converter emitting exactly this
and consumers rejecting it).

validate_references now checks that to_columns covers the to dataset's
primary_key or one of its unique_keys. Coverage rather than exact
equality: a superset of a key still guarantees the join cardinality,
and the databricks converter's _covers_unique_key already applies the
same semantics. Reported as a warning rather than an error because
declared keys may be an incomplete recovery of the dataset's real
keys, and datasets that declare no keys are skipped entirely.

Shape guards keep the semantic check from crashing or misreporting on
documents that already fail schema validation (null unique_keys,
non-list to_columns, flat unique_keys).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: km <kayemkim@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant