Retire the client-side changelog registry refresh - #3760
Merged
Conversation
cotti
force-pushed
the
changelog-retire-client-registry-refresh
branch
from
August 4, 2026 01:50
4cefbe5 to
cc65ecf
Compare
reakaleek
approved these changes
Aug 4, 2026
Mpdreamz
approved these changes
Aug 4, 2026
cotti
force-pushed
the
changelog-retire-client-registry-refresh
branch
from
August 6, 2026 15:30
cc65ecf to
95b64ed
Compare
cotti
changed the base branch from
changelog-registry-reconcile-cli
to
changelog-registry-scrubber-ownership
August 6, 2026 15:34
cotti
added a commit
that referenced
this pull request
Aug 6, 2026
Restacked on the scrubber-ownership chain: the scrubber Lambda owns the
public bundle/{product}/registry.json manifests and the shallow per-tree
maps (#3738), and the client-side refresh machinery is retired (#3760).
The migration command now writes YAML bundle objects only; the S3 events
those creates emit trigger the reconciliation that materializes the
manifests. Tests assert no registry.json is ever PUT.
9 tasks
Contributor
Author
|
Restacked: #3741 was closed with the per-product cutover tooling, so this now sits directly on #3738 (rebased onto its reworked branch — shallow per-tree maps, group reconcile scoped to the bundle tree). Same single commit, no content changes beyond the rebase; 849/849 tests green on the new chain. Now part of stack #3800 (#3738 ← this ← #3794). |
cotti
force-pushed
the
changelog-retire-client-registry-refresh
branch
from
August 10, 2026 16:21
95b64ed to
5695678
Compare
Mpdreamz
approved these changes
Aug 10, 2026
cotti
added a commit
that referenced
this pull request
Aug 10, 2026
Restacked on the scrubber-ownership chain: the scrubber Lambda owns the
public bundle/{product}/registry.json manifests and the shallow per-tree
maps (#3738), and the client-side refresh machinery is retired (#3760).
The migration command now writes YAML bundle objects only; the S3 events
those creates emit trigger the reconciliation that materializes the
manifests. Tests assert no registry.json is ever PUT.
cotti
force-pushed
the
changelog-retire-client-registry-refresh
branch
from
August 10, 2026 19:01
5695678 to
ba57fc6
Compare
cotti
added a commit
that referenced
this pull request
Aug 10, 2026
Restacked on the scrubber-ownership chain: the scrubber Lambda owns the
public bundle/{product}/registry.json manifests and the shallow per-tree
maps (#3738), and the client-side refresh machinery is retired (#3760).
The migration command now writes YAML bundle objects only; the S3 events
those creates emit trigger the reconciliation that materializes the
manifests. Tests assert no registry.json is ever PUT.
The scrubber Lambda is the sole producer of the public registry.json, reconciled from public bucket state on the S3 events every upload already emits (elastic/docs-eng-team#688 Phase 3). Uploads now write YAML objects only; RegistryBuilder and the private-manifest write path are removed, and the amend end-to-end test exercises RegistryReconciler instead.
cotti
added a commit
that referenced
this pull request
Aug 11, 2026
Restacked on the scrubber-ownership chain: the scrubber Lambda owns the
public bundle/{product}/registry.json manifests and the shallow per-tree
maps (#3738), and the client-side refresh machinery is retired (#3760).
The migration command now writes YAML bundle objects only; the S3 events
those creates emit trigger the reconciliation that materializes the
manifests. Tests assert no registry.json is ever PUT.
cotti
force-pushed
the
changelog-retire-client-registry-refresh
branch
from
August 11, 2026 12:54
ba57fc6 to
31d19e6
Compare
cotti
added a commit
that referenced
this pull request
Aug 11, 2026
Restacked on the scrubber-ownership chain: the scrubber Lambda owns the
public bundle/{product}/registry.json manifests and the shallow per-tree
maps (#3738), and the client-side refresh machinery is retired (#3760).
The migration command now writes YAML bundle objects only; the S3 events
those creates emit trigger the reconciliation that materializes the
manifests. Tests assert no registry.json is ever PUT.
cotti
added a commit
that referenced
this pull request
Aug 13, 2026
Restacked on the scrubber-ownership chain: the scrubber Lambda owns the
public bundle/{product}/registry.json manifests and the shallow per-tree
maps (#3738), and the client-side refresh machinery is retired (#3760).
The migration command now writes YAML bundle objects only; the S3 events
those creates emit trigger the reconciliation that materializes the
manifests. Tests assert no registry.json is ever PUT.
cotti
added a commit
that referenced
this pull request
Aug 13, 2026
Restacked on the scrubber-ownership chain: the scrubber Lambda owns the
public bundle/{product}/registry.json manifests and the shallow per-tree
maps (#3738), and the client-side refresh machinery is retired (#3760).
The migration command now writes YAML bundle objects only; the S3 events
those creates emit trigger the reconciliation that materializes the
manifests. Tests assert no registry.json is ever PUT.
cotti
added a commit
that referenced
this pull request
Aug 13, 2026
* Changelog: add temporary migrate-from-web command
* Changelog: test and document migrate-from-web
* Changelog: migrate-from-web writes bundles only (no registry refresh)
Restacked on the scrubber-ownership chain: the scrubber Lambda owns the
public bundle/{product}/registry.json manifests and the shallow per-tree
maps (#3738), and the client-side refresh machinery is retired (#3760).
The migration command now writes YAML bundle objects only; the S3 events
those creates emit trigger the reconciliation that materializes the
manifests. Tests assert no registry.json is ever PUT.
* Changelog: migrate-from-web covers the scope table, --products narrows
Review feedback (#3794): drop the standing config surface. The former
config/migrate-from-web.yml becomes a checked-in table in the command
itself (MigrateFromWebScope.All) — temporary tooling state, added per
rollout wave and deleted with the command. A run covers every product
in the table by default; --products x,y,z narrows it for tests and
pilots. The page→product mapping stays explicit because bundle product
ids appear in no published metadata: page frontmatter carries the site
taxonomy, not bundle ids.
One product's failure no longer aborts the run: each product migrates
independently and the run fails at the end if any of them did.
* FileSystemFactory: extend the mock-temp scope workaround to Windows
MockFileSystem hardcodes its temp path on every OS (C:\temp on Windows,
unix-ified /temp elsewhere) instead of calling Path.GetTempPath(), while
AllowedSpecialFolder.Temp resolves the real temp. The existing workaround
only covered non-Windows, assuming the two coincide there — they don't:
GitHub runners' real temp is under the user profile, so the first test to
write through mockFs.Path.GetTempPath() on Windows (migrate-from-web's
bundle staging) failed scope validation. Add the inner mock's temp as an
explicit root unconditionally; drop once TestableIO#1454 ships.
cotti
added a commit
that referenced
this pull request
Aug 13, 2026
The draft predated the #3738 review rework: RegistryReconciler is now BundleRegistryReconciler and reconciles the bundle/{product}/ tree only, the changelog/ pool manifests remain legacy client-authored pass-through (changelog bundle still enumerates pools through them, RFC #698 replaces that), the registry reconcile/verify operator CLI was dropped with #3741, and the scrubber now also maintains the shallow per-tree folder-to-token maps (consumer side: #3801). Infra bullets match the applied docs-infra#360 IAM (no registry-operator grant, no private ListBucket) and observability as it exists (metric stream to docs-o11y; alerts and runbook tracked in docs-eng-team#692).
cotti
added a commit
that referenced
this pull request
Aug 13, 2026
* Scrubber Lambda owns the public changelog registry via state reconcile Phase 1 of elastic/docs-eng-team#688. The public registry.json was a log of upload operations (client-written, pass-through copied); every known consistency gap followed from that. The scrubber Lambda now derives it from the public bucket's actual state: registry = f(state), never f(event). - Extract the Lambda's top-level handler logic into testable classes in Elastic.Changelog: ScrubberProcessor (batch coalescing by key and group, object-level reconcile with post-write source validation) and RegistryReconciler (delimited/paginated group listing, ETag reuse with amends always recomputed, semantic idempotence, conditional PUT/DELETE with bounded jittered retries on 412/409, newer-schema refusal). Program.cs is now a thin adapter. - Retire the registry pass-through in the same deploy: registry-key events only schedule a group reconcile, so client-authored JSON no longer reaches the public bucket uninspected. - Add a producer (algorithm version) field to the manifest; a mismatch — including legacy pass-through manifests — forces a full metadata recompute and a write even when entries are identical. - Emit per-invocation reconcile metrics as CloudWatch EMF (the Phase 0 observability item that could only land with the reconciler). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add changelog registry reconcile and verify commands Phase 2 of elastic/docs-eng-team#688. The cutover/heal tooling for the Lambda-owned public registry: - `changelog registry reconcile` plans groups (one scope, or the union of both buckets so orphan public groups are covered) and sends one versioned, discriminated reconcile message per group to the scrubber queue — {kind, version, scope, group, correlation_id}, validated through ChangelogKeys on both ends. The CLI never mutates S3; the Lambda stays the public bucket's single writer. --dry-run prints the plan; the non-dry-run path asks for confirmation (--yes for CI). Each run stamps one correlation id and prints a ledger line per group. - On a reconcile message the Lambda performs a full group heal: object-level reconcile over the union of both buckets' listings (copy what's live, delete what isn't), then the group reconcile — recovering lost/DLQ-expired scrub events. Requires the new optional PRIVATE_BUCKET_NAME Lambda env var; malformed messages are rejected to the DLQ where the Phase 0 alarm surfaces them. - `changelog registry verify` is the read-only sibling and cutover gate: compares each public manifest against what a reconcile would write (same listing spec and entry rules by construction) and reports divergence as missing/stale/corrupt/object-divergent, with unsupported schemas reported distinctly. - Fix the manifest ETag wire format: the snake_case policy serialized the producer-side field as "e_tag" while consumers and the documented format read "etag" — recorded ETags were invisible to every consumer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Suppress CA1001 on ChangelogRegistryServiceTests Same suppression RegistryBuilderTests carries: xUnit owns the test class lifetime and TestDiagnosticsCollector needs no disposal in these tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Retire the client-side registry refresh from changelog upload The scrubber Lambda is the sole producer of the public registry.json, reconciled from public bucket state on the S3 events every upload already emits (elastic/docs-eng-team#688 Phase 3). Uploads now write YAML objects only; RegistryBuilder and the private-manifest write path are removed, and the amend end-to-end test exercises RegistryReconciler instead. * Rewrite changelog registry docs for scrubber Lambda ownership The registry docs still described the retired model: client-side refresh, registry pass-through, pre-scrub ETags, a 1 h CloudFront TTL (caching is disabled), and a refresh "skipped for --artifact-type changelog". Documents the reconciler as sole producer, the public-object ETag, convergence semantics, absent-vs-empty manifests, the reconcile message contract, and the registry reconcile/verify operator commands (docs-eng-team#688 Phase 4). * Align registry docs with the merged #3738 rework and #3760 The draft predated the #3738 review rework: RegistryReconciler is now BundleRegistryReconciler and reconciles the bundle/{product}/ tree only, the changelog/ pool manifests remain legacy client-authored pass-through (changelog bundle still enumerates pools through them, RFC #698 replaces that), the registry reconcile/verify operator CLI was dropped with #3741, and the scrubber now also maintains the shallow per-tree folder-to-token maps (consumer side: #3801). Infra bullets match the applied docs-infra#360 IAM (no registry-operator grant, no private ListBucket) and observability as it exists (metric stream to docs-o11y; alerts and runbook tracked in docs-eng-team#692). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Phase 3 of elastic/docs-eng-team#688, stacked on #3738. With the scrubber Lambda as the sole producer of the public bundle manifests and the shallow per-tree maps (#3738), the client-side registry refresh is the last remaining writer in the retired multi-writer model. Keeping it would let a stale private manifest linger (nothing consumes it anymore) and would keep upload logs claiming registry responsibility the CLI no longer has.
Note
Restacked 2026-08-06: this PR previously sat on #3741 (registry reconcile/verify CLI), which was closed after review — the per-product registry cutover tooling went with it. It now stacks directly on #3738, whose rework scopes group reconciliation to the bundle tree and adds the shallow folder→token maps.
What
changelog uploadnow writes YAML objects only: the post-upload refresh block,RegistryBuilder, and the private-manifest write path are gone. The registry materializes from the S3 events the upload already emits. The amend end-to-end test now drivesRegistryReconciler(the production path) for its registry step, and the upload tests assert that noregistry.jsonis ever PUT.Old binaries in the wild keep writing private manifests — harmless: bundle registry-key events only schedule a group reconcile and are never copied to the public bucket.
Important
Do not merge until the Phase 2 cutover checkpoint passes (scrubber Lambda deployed with #3738, queue drained, DLQ empty, and a spot-check that public
bundle/{product}/registry.jsonmanifests carry the reconciler'sproducerfield — see elastic/docs-eng-team#688). Releasing this before the Lambda owns the public registry would leave registries permanently stale.After this releases, the now-dead private
registry.jsonobjects can be deleted (operational step, no code): theirObjectRemovedevents only schedule redundant reconciles.Part of elastic/docs-eng-team#688 (Phase 3).