fix: split publish into siblings-then-bitgo passes

[lokesh-bitgo]

Ticket: WCI-1818

What changes are in this PR

Phase 2 of WCN-1818. Splits the single lerna publish call in both .github/workflows/publish.yml (alpha/beta) and .github/workflows/npmjs-release.yml (prod) into two passes — siblings first, bitgo alone second — using the set-umbrella-publishable.ts toggle script added in Phase 1 (#TODO-phase1-pr-number). This is the change that actually fixes the underlying race condition; Phase 1 only fixed the detection/safety-check logic without changing when publishing happens.

What was the issue

bitgo's prepack script (scripts/generate-bitgo-shrinkwrap.ts) resolves its 89 @bitgo/* sibling packages from the npm registry to build the npm-shrinkwrap.json bundled inside the published bitgo tarball — this is what lets npm install bitgo actually pull in all its siblings instead of shipping an empty shell (WCI-1200).

The problem: lerna publish runs every package's lifecycle hooks (bitgo's prepack included) before uploading any of them. So at the exact moment bitgo's prepack tries to resolve its siblings, none of them are live on the registry yet — they're all still mid-pipeline, exactly like bitgo itself.

This wasn't theoretical — it happened for real. The prod release immediately after the Phase 1-equivalent fix landed (run 30934555125) failed with:

npm error notarget No matching version found for @bitgo/abstract-lightning@^8.2.3.
lerna ERR! lifecycle "prepack" errored in "bitgo", exiting 1

Because this happens before any upload step, zero packages published in that run — not just bitgo.

It also went unnoticed for a while because beta never exercised this code path at all: publish.yml never set BITGO_GENERATE_SHRINKWRAP=true, so the shrinkwrap generator always silently no-op'd there (confirmed directly in a "successful" beta run's own logs: BITGO_GENERATE_SHRINKWRAP not set to "true" — skipping npm-shrinkwrap.json generation.). A green beta build was never meaningful evidence that publishing actually worked.

What we are solving

Making bitgo's siblings genuinely live on the registry before bitgo's own prepack tries to resolve them — without changing what actually gets published, what versions get assigned, or any of the surrounding checks (GPG signing, OSV vulnerability gate, GitHub release creation, Express Docker publish).

What we have done

The core mechanism, same in both workflows:

1. Hold bitgo back: npx tsx ./scripts/set-umbrella-publishable.ts false sets "private": true on modules/bitgo/package.json. Lerna's filterPrivatePkgUpdates filters private packages out before packing (verified directly against lerna 9.0.0's source, node_modules/lerna/dist/commands/publish/index.js), so bitgo's prepack simply doesn't run yet.
2. Publish siblings: normal lerna publish. All 89 siblings publish; bitgo is invisible to this pass.
3. Restore bitgo: set-umbrella-publishable.ts true, under if: always() — runs even if pass 1 failed, so bitgo can never end up permanently stuck private by accident.
4. Publish bitgo: lerna publish again. Only bitgo is left to publish, and its siblings are now genuinely resolvable.

.github/workflows/publish.yml (alpha/beta)

Replaced the single Lerna Publish step with the four steps above (Hold back → Lerna Publish (siblings) → Restore → Lerna Publish (bitgo)). The bitgo pass now also sets BITGO_GENERATE_SHRINKWRAP: true — this workflow never set it before, so this permanently closes the "beta never validates this" gap going forward, not just for one test run. continue-on-error: ${{ inputs.recovery-mode }} (the existing "don't hard-fail on a stuck-package Rekor conflict during recovery" escape hatch) is applied to both new publish steps, since that conflict could now occur in either pass.

.github/workflows/npmjs-release.yml (prod)

Bigger change because versioning and publishing were combined in one command here. Split into:

1. Bump version (!recovery-mode only) — moved the version-bump flags (--sign-git-tag --sign-git-commit --include-merged-tags --conventional-commits --conventional-graduate --yes) onto lerna version alone. Still commits, tags, signs, and pushes to rel/latest exactly as before — just doesn't publish anything yet.
2. The same hold-back → publish siblings (pass 1) → restore → publish bitgo (pass 2) sequence, both steps using lerna publish from-package --yes. Because both the normal path (runs right after the version bump) and the recovery path (runs with no preceding bump, from-package just picks up whatever's on disk and missing from npm) end up calling the exact same from-package command, one shared sequence now covers both paths — previously these were two separate, differently-shaped steps (Publish new version using plain lerna publish, and Publish missing versions (recovery) using from-package).
3. BITGO_GENERATE_SHRINKWRAP: true moved from the old single combined step onto pass 2 specifically (the only pass that actually packs bitgo).
4. Verify recovery published the missing versions (unchanged content, still recovery-only) now runs after pass 2 instead of after the old single recovery step — same relative position in the job otherwise.
5. Nothing else in the job changed: GPG signing config, the OSV vulnerability gate, Verify all packages exist on npm, Verify bitgo package has shrinkwrap metadata, GitHub release creation, and the Express Docker publish steps are all untouched.

Why this is needed

Without this, every prod release that actually reaches the point of resolving bitgo's siblings will fail the same way the real incident did — deterministically, not intermittently, because the siblings genuinely never exist yet at that point in a single combined publish. Phase 1 (already merged/staged separately) made the failure cleaner and more informative when it happens, but did not change whether it happens. This PR is what actually makes npm install bitgo work end-to-end for real consumers going forward.

Considered and explicitly rejected: lerna's --include-private

Lerna has a built-in --include-private <names> flag that publishes a named private package by temporarily removing private from its manifest — which would let bitgo stay permanently private: true in the committed repo with no scripted toggle needed. Rejected because three separate checks in the pipeline enumerate non-private packages to verify they exist/were published:

• the pre-publish existence check (trusted publishing depends on it)
• recovery verification
• beta verification / recovery auto-retry

A permanently-private bitgo would silently stop being covered by all three. The scripted, temporary toggle (flip off → publish → flip back on, all within the same job run, never committed) keeps every one of those checks seeing a normal, publishable package by the time they run.

Correctness review performed before opening this PR

• Traced every if:/continue-on-error/always() interaction by hand across normal-mode-success, normal-mode-pass-1-failure, recovery-mode-success, and recovery-mode-pass-1-failure scenarios in both files — confirmed the restore step fires in every case, and that a genuine pass-1 failure in normal mode still correctly skips pass 2 (fail-fast, same as the original single-step behavior).
• Ran actionlint (with shellcheck) via Docker against both files:

  docker run --rm -v "$PWD:/repo" -w /repo rhysd/actionlint:latest \
    .github/workflows/publish.yml .github/workflows/npmjs-release.yml

  Zero issues in anything changed by this PR — the only 3 warnings reported are pre-existing, in original untouched lines (72/77/81 of publish.yml), unrelated to this change.
• Validated both files as syntactically correct YAML (yaml.safe_load).
• Aligned the new script's invocation path style across both files (npx tsx ./scripts/set-umbrella-publishable.ts, matching publish.yml's existing ./scripts/... convention and npmjs-release.yml's own ./.github/actions/... local-reference convention — that file had no prior npx tsx calls to match against).

What this PR does not cover yet

• No live CI dispatch has been run yet. Everything above is static analysis (linting, manual trace-through) — the actual runtime behavior (does pass 2 really resolve siblings published moments earlier by pass 1? does the always() restore really fire under a real GitHub Actions failure, not just in my reading of the semantics?) has not been exercised for real yet. See "How to validate after merging" below.
• Verify recovery published the missing versions is still recovery-only; a package outside bitgo's dependency tree silently failing to publish in normal mode's pass 1 wouldn't be caught by a dedicated check (only bitgo's own 89 dependencies get implicitly checked, via pass 2's resolution). Confirmed this is a pre-existing gap, not something this PR introduces or worsens — normal mode never had this check before either, since the old single-command lerna publish would hard-fail immediately on any real error. Deliberately left out of scope here; worth its own separate follow-up if desired.
• rel/latest/master branch divergence (orphaned 52.4.2 tags from the original failed release) is still unresolved — a separate, pre-release-time cleanup task, not blocking this PR's review.
• scripts/verify-release.ts's recovery-retry path still doesn't set BITGO_GENERATE_SHRINKWRAP — a known, documented gap from Phase 1, harmless today (beta never shipped a shrinkwrap before this PR's beta change either — though note this PR does now enable it on the main beta publish path, just not on that specific retry script's path), tracked separately.

How to validate this after merging

Static checks (lint, YAML parse, manual trace-through) are already done above; none of them prove runtime behavior. This is the concrete, ordered checklist to actually run — organized by which workflow to dispatch, with exact inputs and exact things to look at.

Step 1 — Publish @bitgo-beta (file: .github/workflows/publish.yml), normal dispatch

Where: GitHub → Actions tab → workflow named "Publish @bitgo-beta" → "Run workflow".
Inputs: recovery-mode = false (default — leave unchecked).
Branch: any feature branch off master (this workflow publishes to the alpha preid/dist-tag whenever github.ref != 'refs/heads/master', so this is safe to run from this PR's own branch after merge, or from master directly for a real beta publish).

What to check in the run's logs, step by step:

• Step "Hold back bitgo umbrella for the siblings publish pass" ran and succeeded, with output Marked modules/bitgo/package.json as private....
• Step "Lerna Publish (siblings)" ran, and its output lists all 89 @bitgo-beta/* (or @bitgo/* on the real beta path) sibling packages being published — bitgo itself should not appear in this step's publish list.
• Step "Restore bitgo umbrella for its own publish pass" ran and succeeded, with output Restored modules/bitgo/package.json to publishable....
• Step "Lerna Publish (bitgo)" ran, and inside it the prepack script's own log lines appear:
  • Resolving 89 workspace siblings as part of the shrinkwrap: (not BITGO_GENERATE_SHRINKWRAP not set to "true" — skipping... — if you see that skip message, this step's env var didn't take effect and something is wrong).
  • Followed by a real npm install --package-lock-only / npm shrinkwrap sequence, ending in Wrote .../modules/bitgo/npm-shrinkwrap.json.
• Step "Verify Publish" passed.

This step alone confirms the "beta never exercises this" blind spot is closed — this is the first time this workflow will have ever actually attempted shrinkwrap generation.

Step 2 — Confirm the actual published package works, for real

Take the version number from step 1's "Lerna Publish (bitgo)" log output, then run these outside CI, on your own machine:

# Confirm the registry itself reports a shrinkwrap for this version
curl -sL "https://registry.npmjs.org/@bitgo-beta/bitgo/<version>" | jq '._hasShrinkwrap'
# expected: true

# Confirm it actually installs and works, in a totally clean directory
mkdir -p /tmp/bitgo-install-check && cd /tmp/bitgo-install-check
npm init -y
npm install @bitgo-beta/bitgo@<version>
ls node_modules/@bitgo-beta/ | wc -l   # expected: 89 (or close, plus any that only exist as transitive deps of siblings)
node -e "require('@bitgo-beta/sdk-core'); console.log('sdk-core loaded OK')"
node -e "require('@bitgo-beta/bitgo'); console.log('bitgo loaded OK')"

• _hasShrinkwrap is true.
• node_modules/@bitgo-beta/ actually contains the sibling packages (not empty).
• Both require() calls succeed with no MODULE_NOT_FOUND.

This is the actual end-to-end proof that WCI-1200 is fixed — not "the workflow didn't error," but "a real npm install from a clean directory works."

Step 3 — Prove the always() restore really survives a failure

Don't just trust the YAML semantics — force a failure and check the aftermath:

How: dispatch Publish @bitgo-beta again, but engineer a failure in the siblings pass (e.g., temporarily point --dist-tag at something that will collide, or dispatch twice in quick succession so the second run hits an already-published-version conflict).

• Confirm the run shows "Lerna Publish (siblings)" as failed (or failed-but-continued, if recovery-mode was also on).
• Confirm "Restore bitgo umbrella for its own publish pass" still shows as having run (visible in the Actions UI step list — it should not be greyed out/skipped).
• Check out the branch/commit that run operated on and confirm modules/bitgo/package.json does not have "private": true left in it.

Step 4 — BitGoJS Release (file: .github/workflows/npmjs-release.yml), dry-run first

Where: GitHub → Actions tab → workflow named "BitGoJS Release" → "Run workflow".
Inputs: dry-run = true, recovery-mode = false.

• Confirm the run completes successfully up through Verify all packages exist on npm.
• Confirm Bump version, Hold back bitgo umbrella..., Publish siblings (pass 1), Restore bitgo umbrella..., Publish bitgo (pass 2) all show as skipped (they're all gated on dry-run == false) — this just confirms the if: gating itself is wired correctly; a dry run does not exercise any of the new logic.

Step 5 — Resolve rel/latest/master divergence (prerequisite, not part of this PR)

Before any real (non-dry-run) prod dispatch: rel/latest currently has an orphaned 52.4.2 version bump from the original incident that master doesn't have, and master has 3 commits rel/latest doesn't have. The verify-back-merge job in npmjs-release.yml will actively block a normal dispatch until this is reconciled. This is tracked separately — do not attempt step 6 until it's resolved.

Step 6 — BitGoJS Release, real dispatch in recovery mode

Inputs: dry-run = false, recovery-mode = true.
Recovery mode specifically because rel/latest already holds the (unpublished) 52.4.2 version bump — a normal-mode dispatch would re-bump past it and orphan those tags further.

• Confirm Bump version is skipped (recovery mode never re-versions).
• Confirm Hold back bitgo umbrella... → Publish siblings (pass 1) → Restore bitgo umbrella... → Publish bitgo (pass 2) all ran, in that order, and all succeeded.
• Confirm "Verify recovery published the missing versions" passed (walks every non-private package, confirms it's reachable on the real npm registry).
• Confirm "Verify bitgo package has shrinkwrap metadata" passed — this is the step that never got the chance to run in the original failed release (job died before reaching it); seeing it pass for real is the final confirmation.
• Repeat Step 2's clean-install check, this time against the real bitgo package (not @bitgo-beta/bitgo), to confirm the fix holds in prod too.

Ticket: WCN-1818

[linear-code]

WCN-1818