fix: rebuild budget covers drift on every owned pod; dead-letter parks at a generation - #36
Merged
Conversation
…a generation The drift-delete path had no rebuild budget on main, sub-agent or toolbox pods. Any defaulter the spec does not pin (a namespace LimitRange with a cpu or memory default, a resource-rewriting webhook) makes podSpecMatchesAgent permanently false, so the pod was deleted and recreated every requeue, one microVM built and destroyed per cycle, with no metric moving. Every terminal or drifted pod now goes through one budget keyed by pod name: four attempts with 0/1/5/30 s backoff, then dead-letter. A dead-lettered pod records the CocoonSet generation and stays parked until a spec edit, which resets its budget; drift alone no longer lifts it, because a permanently rewritten pod would otherwise loop at the slower cadence. The history annotation keeps only pods the spec still names.
…ethods The package created yesterday sat in its own import group in six files. Reconcile and the logr sink's Enabled implement documented interfaces.
… not recreated History entries record the CocoonSet generation they were counted at, so a spec edit resets every budget instead of only the parked pod's, and a Count=4 entry left behind after the fourth delete cannot stamp a fresh generation's pod straight into dead-letter. A missing pod whose name exhausted its budget at the current generation stays absent (main reports Failed, sub-agent slots and toolboxes are skipped), so Kubernetes' terminated-pod GC removing a parked pod no longer restarts the create/fail/park cycle. Comment and doc placement corrected.
…count Count reaching the budget only means the last delete happened; the replacement still has to be created. A parked flag in the history entry now marks the dead-lettered name, the missing-pod gates read that flag, an existing toolbox is always triaged, and an exhausted absent main reports Failed without a timer requeue since only a generation or pod event can change the decision.
…ll matches its spec
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Whole-repo quality round (2026-09-03): every Go file read against the /code Style Self-Check and the judgment lenses; three commits.
LimitRangewith a cpu/memory default, a resource-rewriting webhook) makespodSpecMatchesAgentpermanently false, so the pod was deleted and recreated every requeue (~2 s), one microVM built and destroyed per cycle, and no metric moved. Every terminal or drifted pod now goes through one budget keyed by pod name (rebuild.go): four attempts with 0/1/5/30 s backoff, then dead-letter with theSubAgentDeadLetterevent and counter. A dead-lettered pod records the CocoonSet generation and stays parked until a spec edit (new generation) resets its budget; drift alone no longer lifts it, since a permanently rewritten pod would otherwise loop at the slower cadence.ensureToolboxesreports its pending backoff likeensureSubAgents;Reconcilepicks the shorter. The history annotation prunes pods the spec no longer names (main,[1..replicas], toolboxes) and no longer mutates its input. Metric names are unchanged; their help text now covers every owned pod.podpatch(created yesterday) joins the repo import group in six files; no godoc on theReconcileand logrEnabledinterface methods.podpatch/; docs/cocoonset.md names the current symbols; docs/observability.md describes the budget as it now works.Held with reasons:
podsHibernatedByCRlisting twice per reconcile on the node-pinned + CR-hibernated main path (two in-cache reads; threading a memoized list touches 32 test call sites);RecoveredFromFailurere-firing while sub-agents churn (bounded now that drift converges); a pod label index forlistOwnedPods(one namespace-scoped in-memory walk per reconcile; the repo already has the indexer pattern if density ever asks for it).Gates:
GOWORK=off make lint0 issues on linux and darwin (Reconcile's complexity stays under the gocyclo cap viarebuildDriftedMain);make fmt-checkclean;asl ./...clean on both GOOS;GOWORK=off go test -race -count=1 ./...green. Round diff: 22 files, +239/−222; comment lines +8/−9.