Add temporary changelog migrate-from-web command

[cotti]

Summary

Adds docs-builder changelog migrate-from-web, the deliberately temporary one-off migration command tracked by elastic/docs-eng-team#736 (epic: elastic/docs-eng-team#656). It migrates release notes that were published before the changelog pipeline existed into the S3 bundle store, and will be deleted once the rollout (elastic/docs-eng-team#683) completes.

Design follows the direction set in the review on #3681: plain code, minimal surface, no new schemas/envelopes/frameworks, delete-after-use. The whole feature is one folder (src/services/Elastic.Changelog/Migration/, 3 files) and one CLI method — all marked TEMPORARY. Per the review, there is no config file: the scope table is checked into the command itself (MigrateFromWebScope.All).

Note

Restacked 2026-08-06 onto the registry-ownership chain (#3738 ← #3760 ← this): with the scrubber Lambda owning the public manifests and the client-side refresh retired, the command now writes YAML bundle objects only — the earlier best-effort RegistryBuilder refresh was removed, and tests assert no registry.json is ever PUT.

How it works

1. Table scope, everything by default — a run covers every product in the checked-in scope table (MigrateFromWebScope.All: product id → owner/repo/path/ref/cutoff); --products x,y,z narrows it for tests and pilots. The page→product mapping stays explicit because bundle product ids appear in no published metadata (page frontmatter carries the site taxonomy, not bundle ids). --versions narrows a run further; versions above a product's cutoff are always skipped (they belong to the live pipeline). One product's failure doesn't block the others; the run fails at the end if any product did.
2. Fetch — downloads the release-notes Markdown from raw.githubusercontent.com at the pinned commit SHA in the table.
3. Map — parses ## {version} sections (typed ### … subsections → BundledEntry with PR references extracted; prose → bundle description, preserved verbatim) into the existing Bundle shape that changelog upload publishes. No new fields or schemas.
4. Create-only upload — writes bundle/{product}/{version}.yaml with If-None-Match: *. Existing keys are skipped (identical vs. different content is distinguished via ETag), 412 races are reported as skips, nothing is ever overwritten. No registry writes: the scrubber Lambda materializes bundle/{product}/registry.json and the shallow per-tree maps from the S3 events these creates emit (Move public changelog registry ownership to the scrubber Lambda #3738; client-side refresh retired in Retire the client-side changelog registry refresh #3760).
5. Run report — one line per key (created / would-create / skipped / failed + reason + ETag), printed as a Markdown table ready to paste into the tracking issue.

Content source: Markdown at a pinned ref, not live HTML

Per the issue's preference, the command fetches the raw Markdown that backed the published pages instead of scraping elastic.co HTML: it is the authoritative source (the site is rendered from it), structured (headings/bullets/PR links survive intact), and a pinned SHA makes runs reproducible. For the EDOT Java pilot the table pins 9a61ce4f, the last hand-authored state of docs/release-notes/index.md before elastic/elastic-otel-java#1023 switched that repo to native bundle YAMLs.

Create-only mechanism

HEAD (GetObjectMetadata) first — so skips can report the existing object's ETag and whether content is identical — then a conditional PutObject with If-None-Match: * as the actual race guard (a concurrent create surfaces as HTTP 412 and is reported as a skip). This mirrors the machinery from the closed #3715 but implemented locally in the temporary service to keep shared surface area at zero.

Running the EDOT Java pilot

# credential-free dry run (parses, maps, reports; no S3 access at all)
docs-builder changelog migrate-from-web edot-java --dry-run

# dry run against the real bucket (distinguishes would-create from skipped)
docs-builder changelog migrate-from-web edot-java --dry-run --s3-bucket-name <bucket>

# the real thing
docs-builder changelog migrate-from-web edot-java --s3-bucket-name <bucket>

Actual dry-run output (abridged — 13 releases parsed, 1.0.0 through 1.10.0):

### Run report: changelog migrate-from-web

- product: `edot-java`
- source: `elastic/elastic-otel-java@9a61ce4faaf08e272c433a083bcc6f0e96d80e0a` `docs/release-notes/index.md`
- cutoff: `1.10.0`
- mode: dry-run (no S3 writes)

| key | outcome | etag | detail |
|---|---|---|---|
| `bundle/edot-java/1.0.0.yaml` | would-create | `0d143b466fb7539fb572655fd6405581` | no bucket specified; existence not checked |
| ... | | | |
| `bundle/edot-java/1.9.0.yaml` | would-create | `fe5fc47886d346de106d7b0057eaea49` | no bucket specified; existence not checked |

totals: would-create 13

Temporary

The command, its service folder (scope table included), and docs page are all explicitly marked TEMPORARY (help text, XML docs, docs page warning) and will be removed after elastic/docs-eng-team#683.

Test plan

• Parser: realistic EDOT Java-style Markdown fixture → bundle mapping (versions, release dates, entry types, PR-link variants incl. bare #123 refs, prose preservation, unrecognized-subsection warnings, serialized YAML round-trips through ReleaseNotesSerialization)
• Create-only: first run creates every in-scope key with If-None-Match: *; second run over the same scope is a no-op (all skips, zero PUTs)
• Existing key with different content is skipped, never overwritten; concurrent-create 412 reported as skip
• Failure reporting: per-key failed with reason, non-zero exit
• Cutoff and --versions scope filtering
• Dry-run with and without bucket; no writes either way; no S3 access at all without a bucket
• All fixtures in-memory (fake S3 client, stub HTTP handler, mock filesystem) — no network in tests
• Never writes a registry.json (asserted after the restack onto Retire the client-side changelog registry refresh #3760)
• dotnet format clean · ./build.sh build --skip-dirty-check ✓ · AOT publish zero warnings ✓ · Elastic.Changelog.Tests 877/877 on the restacked chain ✓ · cli-schema.json regenerated and verified ✓

Refs: elastic/docs-eng-team#736, elastic/docs-eng-team#656, elastic/docs-eng-team#683

[Mpdreamz]

LGTM, but I think we can do without yet another config?

Just migrate everything with a --products x,y,z flag for tests.

[cotti]

Done in efab4d6 — config/migrate-from-web.yml is gone. The scope moved into the command itself as a checked-in table (MigrateFromWebScope.All), so a bare changelog migrate-from-web covers everything the migration knows how to source, and --products x,y,z narrows it for tests/pilots. One product failing no longer aborts the rest of the run.

One thing the table deliberately keeps that a pure flag can't carry: the page→product mapping (owner/repo/path/pinned ref/cutoff per product). Bundle product ids appear in no published metadata — page frontmatter carries the site taxonomy (cloud-serverless, observability, edot-sdk on the EDOT Java page), not bundle ids — so deriving the mapping automatically isn't possible. Since the table is code inside the TEMPORARY command, it dies with the command after the rollout (elastic/docs-eng-team#683), and adding a product per wave is a small reviewed PR rather than standing config.

Also rebased onto the current base (the add/add conflict with scrubber-allowlist in ChangelogCommand.cs is resolved), so CI should be running against a fresh tree now.