Skip to content

ci: make the DOOP dataset manifest depend on the data, not the path - #1307

Merged
justinjoy merged 1 commit into
mainfrom
fix/1297-doop-manifest-path
Sep 3, 2026
Merged

ci: make the DOOP dataset manifest depend on the data, not the path#1307
justinjoy merged 1 commit into
mainfrom
fix/1297-doop-manifest-path

Conversation

@justinjoy

Copy link
Copy Markdown
Collaborator

Refs #1297. Stacked on #1298 (fix/1294-manifest-collation) — both change the same line, so basing on main would guarantee a conflict. Merge #1298 first.

The defect

run-doop-perf-gate.sh hashed sha256sum "$data_dir"/*.facts. sha256sum prints <hash> <path>, and nothing stripped the directory — so one directory gave three hashes:

relative    871d21c0…
absolute    2887c99e…
trailing /  01e931b8…

Reported as DOOP dataset manifest … != pinned …, which reads as dataset substitution — the same false accusation #1294 removed from a different cause, in the same message.

The fix, and why the obvious one wasn't enough

( CDPATH= cd -- "$1" && sha256sum -- *.facts | LC_ALL=C sort -k2 | sha256sum | awk '{print $1}' )

Stripping the prefix afterwards was my first fix. GNU sha256sum escapes a path containing a backslash by prefixing the whole line, which a prefix strip can't remove — so two spellings of a backslash-containing directory still disagreed.

CDPATH= is load-bearing. cd consults CDPATH for a relative argument — which $data_dir is by default — and on a hit it prints the resolved path into the command substitution and reaches a different directory. With CDPATH pointing at anything containing a doop, the function returned that directory's hash and exited 0: a manifest whose purpose is proving the benchmark ran on pinned data, certifying data it never read.

-- covers option-shaped names; it does not cover a bare -, which bash converts to $OLDPWD after option parsing. The comment says so rather than claiming otherwise.

Compatibility

This changes the computed value. Nothing in-repo pins WIRELOG_DOOP_DATASET_MANIFEST_SHA256 — it's read from the environment and the gate skips when unset — so there's nothing here to recompute. But an operator holding a value recorded under the old formula sees a mismatch on the first run. The failure message now says the pin is stale rather than the data wrong, and names where the correct value already lives (downstream-matrix-oracles.tsv's files: field). CHANGELOG carries the same.

manifest_for_doop deliberately untouched

Its output is pinned in downstream-matrix-oracles.tsv; moving it inside a change premised on manifests not moving would be self-contradictory. The two agree on a canonical path and diverge on trailing slashes, dotfiles and symlinks — the self-test asserts the agreement and pins the divergences, so unifying them has to be deliberate. That unification is the shared helper the issue asks for.

Three defects in my own test, all one cause

defect why it passed
backslash in the leaf both spellings carried it, sha256sum escaped both identically — reverting to the broken implementation passed the assertion named for that bug
-x.facts in a two-spelling fixture dropping -- broke both sides; two empty strings compared equal
'$data' single-quote interpolation a quote in TMPDIR killed both subshells; two empty strings compared equal

Each was built from the shape of the description rather than the mechanism of the defect. Each is now built from the mechanism and killed by mutation.

Validation

  • 11 assertions, linux-gated (the collation fixture can't discriminate where en_US is codepoint order)
  • meson test -C build: 306 Ok / 0 Fail / 12 Skipped
  • Mutations killed: drop CDPATH= (2 named assertions), drop sha256sum --, revert to the prefix-strip form, drop LC_ALL=C, name-dropping. manifest_for_dir correctly survives; cd -- survives and is documented as out of scope.
  • Hostile environments all exit 0: nine TMPDIR shapes (space, quote, #, *, [, \, $, dot, relative), CDPATH=., CDPATH=/tmp, CDPATH=.:/tmp:/var, OLDPWD=/etc

Also fixes the SIGPIPE twin in check-manifest-collation.sh: locale -a | grep -Fxq reported the locale absent because grep exits first, and that gate's skip is exit 0 — so on a locales-all host it reported OK having asserted nothing.

Four review rounds; final verdicts Reviewer and Critic both approving.

run-doop-perf-gate.sh hashed `sha256sum "$data_dir"/*.facts`.  sha256sum
prints "<hash>  <path>", and nothing stripped the directory, so the
manifest embedded whatever $data_dir expanded to: relative, absolute and
trailing-slash spellings of one directory produced three different
hashes.  The mismatch surfaces as "DOOP dataset manifest ... != pinned",
which reads as dataset substitution -- the same false accusation #1294
removed from a different cause, in the same message.

The manifest is now computed from inside the directory, so no path is
ever embedded rather than embedded and then stripped:

    ( CDPATH= cd -- "$1" && sha256sum -- *.facts | LC_ALL=C sort -k2 ... )

Stripping the prefix afterwards was the obvious fix and is not enough:
GNU sha256sum escapes a path containing a backslash by prefixing the
whole line, which a prefix strip does not remove, so two spellings of a
backslash-containing directory still disagreed.

CDPATH= is load-bearing rather than hygiene.  `cd` consults CDPATH for a
relative argument -- which $data_dir is by default -- and on a hit both
prints the resolved path, landing it inside the command substitution, and
reaches a different directory.  With CDPATH pointing at anything that
also contains a `doop`, the function returned that directory's hash and
exited 0: a manifest whose purpose is proving the benchmark ran on the
pinned data, certifying data it never read.  `--` covers option-shaped
names; it does not cover a bare `-`, which bash converts to $OLDPWD after
option parsing, and the comment says so.

The expression is a function so it can be exercised.  It could not be
tested before without running the whole benchmark, which is why the
collation gate from #1294 could only string-match this site.

Scope: this changes the computed value.  Nothing in this repository pins
WIRELOG_DOOP_DATASET_MANIFEST_SHA256 -- it is read from the environment
and the gate skips when unset -- so there is nothing here to recompute,
but an operator holding a value recorded under the old formula will see a
mismatch on the first run.  The failure message now says so and names
where the correct value already lives, because a fix that re-creates the
accusation it removes is worse than the bug.  CHANGELOG carries the same.

manifest_for_doop in run-downstream-matrix.sh is deliberately untouched.
Its output is pinned in downstream-matrix-oracles.tsv, and moving it
inside a change premised on manifests not moving would be
self-contradictory.  The two agree on a canonical path and diverge on
trailing slashes, dotfiles and symlinks; the self-test asserts the
agreement and pins the divergences, so unifying them has to be
deliberate.  That unification is the shared helper #1297 asks for.

Three defects in the self-test are worth recording, because they share a
cause.  The backslash assertion put the backslash in the leaf, so both
spellings carried it, sha256sum escaped both identically, and reverting
to the broken implementation passed the assertion named for that bug.
The option-shaped fixture was first placed where both sides of an
equality broke together, so two empty strings compared equal.  And the
locale assertion interpolated a path into single quotes, so a TMPDIR
containing a quote killed both subshells and the empty results compared
equal.  Each fixture was built from the shape of the description rather
than the mechanism of the defect, and each is now built from the
mechanism and killed by mutation.

Also fixed here: the SIGPIPE twin in check-manifest-collation.sh, where
`locale -a | grep -Fxq` reports the locale absent because grep exits
first, and that gate's skip is exit 0 -- so on a locales-all host it
reported OK having asserted nothing.

Refs #1297.
@justinjoy
justinjoy force-pushed the fix/1297-doop-manifest-path branch from d6a850f to 2faf99b Compare September 3, 2026 01:02
@justinjoy
justinjoy enabled auto-merge (rebase) September 3, 2026 01:03
@justinjoy
justinjoy merged commit f9e72b2 into main Sep 3, 2026
26 checks passed
@justinjoy
justinjoy deleted the fix/1297-doop-manifest-path branch September 3, 2026 02:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant