Skip to content

fix(atenet): add router health hysteresis - #700

Open
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:health-hysteresis
Open

fix(atenet): add router health hysteresis#700
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:health-hysteresis

Conversation

@orangeCatDeveloper

Copy link
Copy Markdown
Contributor

Fixes #616

Summary

updateComponentHealth set Healthy directly from each probe result, so a single 500ms blip to Envoy's admin port or the Kubernetes API flipped /statusz unhealthy for one polling interval and paged any monitoring built on it. A component now flips unhealthy only after 3 consecutive probe failures; a single success restores it and resets the streak. A component that has never succeeded stays unhealthy (zero value), and message/last_failure/failure_count still update on every probe, so individual flaps remain observable — only the healthy boolean gains hysteresis. The health check is observability-only, so no traffic behavior changes.

Test plan

  • Unit: threshold enforcement, streak reset on recovery, never-healthy zero-value boundary; plus an integration-style test driving rh.check() with a flapping Envoy probe and asserting /statusz holds healthy through 2 transient failures, flips on the 3rd, and recovers on success. Package tests pass with -race.

  • kind cluster: deployed the router with this change and ran the e2e suites — demo, example, identity, networking, networkpolicy, and parking pass. The metrics suite fails identically on a clean-main deployment (pre-existing router-metrics-export issue), confirmed by control experiment.

  • Tests pass

  • Appropriate changes to documentation are included in the PR (none needed)

@orangeCatDeveloper
NekoPunch (orangeCatDeveloper) force-pushed the health-hysteresis branch 4 times, most recently from a7808f1 to 611b834 Compare August 6, 2026 17:02
@orangeCatDeveloper
NekoPunch (orangeCatDeveloper) force-pushed the health-hysteresis branch 9 times, most recently from 42d66bb to f568d02 Compare August 18, 2026 01:59
@orangeCatDeveloper
NekoPunch (orangeCatDeveloper) force-pushed the health-hysteresis branch 5 times, most recently from 9bceb18 to 3859f13 Compare August 25, 2026 03:32
@orangeCatDeveloper NekoPunch (orangeCatDeveloper) changed the title atenet: add router health check hysteresis fix(atenet): add router health hysteresis Aug 25, 2026
@orangeCatDeveloper
NekoPunch (orangeCatDeveloper) force-pushed the health-hysteresis branch 2 times, most recently from 2e52ad6 to 0bb5521 Compare August 27, 2026 07:09
A transient probe failure made statusz unhealthy for one interval and paged monitoring. Require three consecutive failures, while one success restores health and resets the streak. Continue updating diagnostics on every probe so flaps remain visible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P3] Health checker has no hysteresis — single transient probe failure flips status

1 participant