Real install smoke via polylane.com right after each release

[justinhelmer]

Every published release is now immediately installed for real through the production installer (polylane.com/install) in a throwaway HOME — on both verification paths — with a Slack alert on failure. A release that breaks the installer pages within minutes instead of surfacing at the daily cron or from users.

What & why

Prevention item 2 of 2 from the v0.2.20 incident RCA (15m of broken installs; caught by a manual pre-deploy smoke, not by CI — every suite stubs curl, so only a real download exercises the release↔installer contract). Item 1, the daily backstop with the same real-install legs in install-bytes.yml, is polylanedotcom#178.

Changes

• New reusable workflow install-smoke.yml (workflow_call + workflow_dispatch): fetches the production installer, runs a real install pinned to the given version in a fresh HOME (POLYLANE_NO_PATH=1 DO_NOT_TRACK=1 … --no-setup, piped), asserting sha256 verified, a completed install, and the sign-in stop; then repeats it with api.github.com blocked, proving the release ships a working checksums.txt. Failure posts to #polylane-notifications via the same 1Password → Slack pattern as polylanedotcom's install-bytes alert.
• release.yml calls it (needs: release, secrets: inherit) with the just-published version.

Decisions

• workflow_call from release.yml, not on: release: the release is created with the default GITHUB_TOKEN, and events caused by that token never trigger other workflows — an on: release version would silently never run.
• Non-blocking by design: the release is already published when the smoke runs, so the job's value is fast detection + an alert carrying the mitigation (fix the asset / cut a corrected release), not gating.
• Alert path assumption: posting uses this repo's OP_SERVICE_ACCOUNT_TOKEN against the same 1Password environment install-bytes uses; whether this repo's service account can read it is unverified from here — the test_alert dispatch input exists to validate exactly that (human/receipt-gated below).

Validation

• Smoke-script logic dry-run locally against the real production installer pinned to v0.2.20: all three assertions pass (sha256 verified, install completed, sign-in stop) — the exact commands the workflow runs.
• YAML lint clean on both workflow files.
• Post-merge receipts (will be posted here): a workflow_dispatch run with version=0.2.20 green on both legs, and a test_alert=true dispatch proving Slack delivery (validates the OP-environment access assumption). Human-gated final receipt: the next release cut shows the install-smoke job green in its release run.

🤖 Generated with Claude Code

[claude]

Reviewed. No blocking issues found in the diff (.github/workflows/install-smoke.yml +149, .github/workflows/release.yml +16). Verified head SHA 91697dd against the PR API exactly, base not stale, diff stats match claimed (+165/-0, 2 files).

Correctness, traced against the live production install.sh:

• Trigger is correct: install-smoke.yml is workflow_call'd from release.yml's new install-smoke job (needs: release), not release: published — right call, since a release created with the default GITHUB_TOKEN wouldn't fire that event.
• Hits the real https://polylane.com/install endpoint and runs the actual production installer end-to-end (confirmed by fetching the live script) — not a mock.
• Would have caught #176: traced the exact digest-lookup regression class through the live install.sh (lines ~316-395) on both the primary API-digest path and the checksums.txt-fallback path (forced via a PATH-shimmed curl blocking api.github.com) — both assertions fail loudly (sha256 mismatch → no "sha256 verified" string → smoke step exits 1) on that failure mode.
• Fails loudly: every assertion is a hard exit 1; a Slack alert to #polylane-notifications on failure() includes actionable mitigation text and itself asserts the POST succeeded.
• Tests/lint/typecheck all green on head (335/335 tests, lint clean, typecheck clean, after the repo's own documented codegen step). CI (3x Test & typecheck) green. Author is justinhelmer (human) — self-approval pattern doesn't apply.

Non-blocking nits:

1. install-smoke.yml:64 — the outer curl fetching the installer script itself has no --retry/timeout, unlike every other network call in this workflow; a one-off blip there pages the team as "installer endpoint unreachable" indistinguishable from a real outage.
2. install-smoke.yml:108-115 — 1Password CLI installed via sudo with no checksum/signature verification (mirrors an existing pattern in the sibling install-bytes.yml, not new here — worth a follow-up ticket across both).
3. install-smoke.yml:39-40 — permissions: at workflow level vs. release.yml's job-level convention; cosmetic only.
4. The new workflow hasn't been exercised by any CI run yet (only fires post-release or via manual dispatch) — the PR body's promised post-merge dispatch receipts (a real version= run and a test_alert=true run) are still outstanding; worth confirming those land before considering this fully proven in production.

---

Generated by Claude Code

[justinhelmer]

Addressed all four non-blocking nits (amended into the single commit):

1. Installer fetch resilience — the polylane.com/install fetch now carries --retry 3 --connect-timeout 10 -m 60, so a one-off blip can't page as an outage.
2. 1Password CLI verification — the op zip download is now pinned by sha256 (ONEPASSWORD_CLI_SHA256, checked with sha256sum -c before install; version and hash bump together). The sibling pattern in polylanedotcom's install-bytes.yml stays as a follow-up there, as suggested.
3. Permissions placement — moved permissions: contents: read from workflow level to job level, matching release.yml's convention.
4. Dispatch receipts — acknowledged as outstanding by design (the workflow only runs post-release or by dispatch): immediately after merge I'll run a version=0.2.20 dispatch and a test_alert=true dispatch and post both run links here; final receipt is the install-smoke job on the next real release cut.

[claude]

Re-reviewed at new head 20b94d395af958be45ba686d7aa016731fcfefe2 (base still cbca730..., not stale). Diff since my last review is scoped to exactly the 4 nits, nothing else — .github/workflows/install-smoke.yml only, +9/-5, release.yml untouched.

All 4 addressed, verified concretely:

1. Retry/timeout: install-smoke.yml:64 now curl -fsSL --retry 3 --connect-timeout 10 -m 60 https://polylane.com/install -o /tmp/install.sh — fixed.
2. 1Password CLI checksum: new ONEPASSWORD_CLI_SHA256 env + sha256sum -c - check before install. I independently downloaded the real op_linux_amd64_v2.38.0-beta.01.zip from cache.agilebits.com and computed its sha256 myself — it matches the pinned value (ab5617d2...4315857) exactly. Real checksum, not a placeholder.
3. Permissions scope: permissions: contents: read moved from workflow-level to job-level under jobs.smoke, matching release.yml's existing per-job convention. Confirmed via YAML parse.
4. Dispatch receipts: no code change claims this is solved — PR body still correctly describes the workflow_dispatch receipts and the next-release confirmation as outstanding post-merge follow-up. Nothing broken.

Checks on the new head: CI green (3x Test & typecheck, 20.x/22.x/24.x). Locally: lint clean, typecheck clean, 340/340 tests pass (after npm run codegen), YAML valid.

No new issues found. This resolves all four prior nits.

---

Generated by Claude Code

[claude]

LGTM: all four non-blocking nits from the earlier review are addressed at head 20b94d395af958be45ba686d7aa016731fcfefe2 and the re-review found no new issues. Posting the LGTM: verdict token at Justin's request, since the re-review above omitted it and the auto-approve workflow keyed on it did not fire.

---

Generated by Claude Code

[github-actions]

Auto-approved: Claude reviewed this PR and posted an LGTM verdict (see its review). A repo admin enabled this via the auto-approve workflow.

---

Generated by Claude Code

[justinhelmer]

Post-merge receipts:

• Real smoke pinned to v0.2.20: run 32786301321 — green on both legs (API-digest verification and API-blocked checksums.txt fallback) through the production installer.
• Alert-path validation (test_alert=true): run 32786303661 — both smoke legs green, but the Slack post step failed as the PR's flagged assumption predicted: op run → failed to retrieve environment variables (exit 9). This repo's OP_SERVICE_ACCOUNT_TOKEN service account does not have access to 1Password environment 4r6j7zk3wmoppaotpeds6xg63m (nominal's prod env carrying SLACK_REPORTER_TOKEN). Until that grant is made in 1Password (human-gated), a real smoke failure still fails the release workflow run loudly — but no Slack page is delivered. Re-validate after granting with: gh workflow run install-smoke.yml -f version=0.2.20 -f test_alert=true.
• Remaining human-gated receipt: the install-smoke job green in the next real release cut's workflow run.

[justinhelmer]

Resolved: Slack alerting removed by design (see #76)

Outcome. This repo is public, and paging #polylane-notifications requires the SLACK_REPORTER_TOKEN from the 1Password "Prod" environment. Granting a public repo's CI service account access to Prod would expose the whole prod secret set, so that access is withheld by design (decision thread). Rather than leave a permanently-failing alert step, the Slack path is removed in #76.

What was verified along the way. The failure was cli-specific and purely an access difference: the identical op run --environment 4r6j… succeeds from polylanedotcom's private workflow (control run) and fails only here. That difference is exactly the prod-secret exposure we're declining to create — so the fix is to remove the alert, not to grant the access.

Net design. Detection stays here (a broken release fails install-smoke as a red check on the release run, within minutes); Slack paging lives in polylanedotcom's private install-bytes backstop, which runs the same real-install legs daily and holds Prod access legitimately. No prod credential ever sits in this public repo.

[justinhelmer]

Superseded by #76: removing the Slack alert path (public repo must not hold prod secrets — access withheld by design). Detection stays as the red-check install-smoke run; paging stays in polylanedotcom's private backstop. See the updated follow-up comment above.