Skip to content

fix(reproducibility): honest verdicts for four controls (#445, #431, #430, #432) - #454

Open
mlieberman85 wants to merge 1 commit into
darnitdevorg:mainfrom
mlieberman85:038-repro-false-pass
Open

mlieberman85 wants to merge 1 commit into
darnitdevorg:mainfrom
mlieberman85:038-repro-false-pass

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Closes #445, #431, #430, #432.

Four issues filed as one family: "controls that pass on artifact presence rather than artifact quality." Reading the code before writing the spec showed that framing holds for two of them and not the other two, so the fixes differ.

Verdict strength (#445, #431)

RE-03.01 BitForBitReproducible granted a dispositive PASS when SOURCE_DATE_EPOCH, reprotest or diffoscope appeared anywhere in any workflow file. On psf/requests, one occurrence in publish.yml produced PASS at confidence 0.8. Nothing was built; nothing was compared.

Those signals are evidence of intent, not achievement -- a build can set SOURCE_DATE_EPOCH and still embed absolute paths, non-deterministic ordering, or a timestamp from elsewhere. Signals found now produce WARN naming both the signal and the gap.

RE-01.02 BuildEnvDeclared granted PASS when any of eight filenames existed, so a Dockerfile reading FROM alpine:latest satisfied it. Container files are now read.

FROM line Verdict
alpine@sha256:... pinned
scratch pinned
builder (a stage alias) not an image
alpine:latest unpinned
python:3.12-slim-bookworm unpinned
$BASE_IMAGE unpinned

Only a digest pins. A tag is mutable -- 3.12-slim-bookworm is rebuilt regularly and resolves to different bytes over time, which is the property this control claims to establish. Declarations that pin by construction (flake.nix, .tool-versions, .python-version, .nvmrc) are untouched, and a stronger declaration still wins.

Stage aliases are collected before classification. Treating FROM builder as an unpinned image would produce a spurious finding on every multi-stage Dockerfile, including this repository's own.

Detection coverage (#430, #432)

repro_hermetic_build was already careful about its PASS -- it requires a verified Witness attestation with a clean network log, a Nix flake build, or Bazel with a blocking flag, and explicitly rejects a mere mention of witness run in CI text. Its problem was coverage.

_SUSPICIOUS_PATTERNS held seven entries and omitted the Go, Rust and Ruby installers. A version-pinned go install pkg@v1.2.3 stays a violation -- this control's subject is hermeticity, not determinism -- but the message names the pin so a pinned fetch is distinguishable from a floating one.

Compiler flags get their own message. Reporting -march=native through "Possible live network fetches" would be a false statement about what was found. -O3 is deliberately excluded: at a fixed toolchain it is deterministic, and the non-determinism #432 describes comes from the unpinned toolchain.

The propagation worth reviewing

_detect_strong_hermeticity_signal gates its Nix path on RE-01.02 == "PASS". Tightening RE-01.02 therefore withdraws that signal from repositories whose container declaration is unpinned: they lose an RE-02.01 PASS without having changed anything.

That is intended. The gate's stated premise is that "a bare flake.nix that isn't the project's confirmed, declared build environment isn't a strong signal on its own" -- once RE-01.02 stops confirming it, keeping the PASS keeps a verdict whose justification is gone. The result now says the signal was withheld because of RE-01.02, so the cause is visible rather than inferred.

Verification

Re-running the nine repositories used to find these issues produced three verdict changes, each explained by a requirement:

repo change why
cosign RE-01.02 PASS -> WARN golang:1.27.1 is a tag
scorecard RE-02.01 WARN -> FAIL go install, pin named from a real commit SHA
requests RE-03.01 PASS -> WARN the case #445 was filed about

59 new tests; 3266 pass. The corpus expectation is written by hand rather than captured -- this change's claim is "these specific verdicts changed", which a reviewer should be able to read and disagree with. (Feature 037 needed a captured baseline because its claim was the opposite, and learned that a stored golden of control statuses is not portable across environments. This corpus avoids that by disabling witness verification, which removes the only network call in the plugin.)

A limitation this surfaced, not fixed here

darnit does not fail its own RE-01.02 check, despite packaging/container/Dockerfile using FROM python:3.12-slim-bookworm. It has a root .python-version, which is inherently pinned and checked first, so the Dockerfile is never inspected.

That follows from the spec's "a stronger declaration wins" rule and is implemented as specified, but it means any Python project with a .python-version never has its container file checked, however floating its base image. That narrows #431's reach for exactly the ecosystem darnit is written in. I did not change it silently -- it wants either an issue or a spec amendment, and the argument that the weakest declaration should govern seems at least as strong.

Scope held

.devcontainer and Vagrantfile are content-dependent in principle and still judged by presence; widening them is detection work of the same shape as #433 and #446. repro_provenance_exists has the same presence-based shape as #445, is covered by none of these four issues, and is filed separately as #453 rather than bundled in.

…#445, darnitdevorg#431, darnitdevorg#430, darnitdevorg#432)

Four issues filed as one family. Reading the code first showed the framing
held for two of them and not the other two, and the fixes differ accordingly.

Verdict strength
----------------

RE-03.01 (BitForBitReproducible) returned a dispositive PASS when the string
SOURCE_DATE_EPOCH, reprotest or diffoscope appeared anywhere in any workflow
file. Observed on psf/requests: one occurrence in publish.yml produced PASS
at confidence 0.8. Nothing was built and nothing was compared. Those signals
are evidence of intent, not of achievement -- a build can set
SOURCE_DATE_EPOCH and still embed absolute paths, non-deterministic ordering,
or a timestamp from elsewhere. Signals found now produce WARN naming both the
signal and the fact that reproducibility was not verified.

RE-01.02 (BuildEnvDeclared) returned PASS when any of eight filenames
existed, so a Dockerfile reading `FROM alpine:latest` satisfied it. Container
build files are now read. Only a digest pins: every tag is mutable, including
specific-looking ones such as `python:3.12-slim-bookworm`, which is rebuilt
regularly and resolves to different bytes over time. Declarations that pin by
construction -- flake.nix, .tool-versions, .python-version, .nvmrc -- are
unchanged, and a stronger declaration still wins.

Stage references are resolved before classification. A multi-stage build
introduces names with `AS` and later refers back to them, and treating
`FROM builder` as an unpinned image would produce a spurious finding on every
multi-stage Dockerfile, including this repository's own.

Detection coverage
------------------

repro_hermetic_build was already careful about its PASS -- it requires a
verified Witness attestation with a clean network log, a Nix flake build, or
Bazel with a blocking flag, and explicitly rejects a mere mention of
`witness run` in CI text. Its problem was coverage.

_SUSPICIOUS_PATTERNS held seven entries and omitted the Go, Rust and Ruby
installers. `go install`, `go get`, `cargo install` and `gem install` are now
violations. A version-pinned `go install pkg@v1.2.3` remains a violation --
this control's subject is hermeticity, not determinism -- but the message
names the pin, so a pinned fetch is distinguishable from a floating one.

Nothing detected compiler flags that make output depend on the build host.
-ffast-math, -march=native and -mtune=native now produce a finding under
their own message; reporting them through "Possible live network fetches"
would have been a false statement about what was found. -O3 is deliberately
excluded: at a fixed toolchain it is deterministic, and the non-determinism
darnitdevorg#432 describes comes from the unpinned toolchain.

Propagation
-----------

_detect_strong_hermeticity_signal gates its Nix path on RE-01.02 having
PASSED. Tightening RE-01.02 therefore withdraws that signal from repositories
whose container declaration is unpinned, and they lose an RE-02.01 PASS
without having changed. That is deliberate -- the gate's premise is that
RE-01.02 confirmed the declared environment, and it no longer does -- and the
result now says the signal was withheld because of RE-01.02 rather than
leaving the operator to infer the flake went missing.

Verification
------------

Re-running the nine repositories used to find these issues produced three
verdict changes, each explained by a requirement: cosign RE-01.02 PASS->WARN
(golang:1.27.1), scorecard RE-02.01 WARN->FAIL (go install, pin named from a
real commit SHA), requests RE-03.01 PASS->WARN.

59 new tests. The corpus expectation is written by hand rather than captured:
the claim here is "these specific verdicts changed", which a reviewer should
be able to read and disagree with.

Closes darnitdevorg#445
Closes darnitdevorg#431
Closes darnitdevorg#430
Closes darnitdevorg#432
Refs darnitdevorg#453

Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

repro_bit_for_bit grants a dispositive PASS on SOURCE_DATE_EPOCH appearing in a workflow file

1 participant