feat: native gRPC multi-backend failover (internal CI) - #2
Open
songzhendong wants to merge 5 commits into
Open
Conversation
songzhendong
force-pushed
the
feature/python-multi-backend
branch
2 times, most recently
from
August 25, 2026 02:57
2220f37 to
a662eb9
Compare
One channel for the process lifetime with C-core pick_first over comma-separated collector addresses (Node native failover analogue). Single address stays plain host:port; multi uses ipv4:/ipv6: (mixed families via IPv4-mapped). Multi hostnames expand once at channel build. pick_first shuffleAddressList is on; target / default_authority stay in config order. Channel options: HTTP proxy off, no keepalive, reconnect backoff capped at 30s, service_config retries only reportInstanceProperties. Skip reports until READY. Unary and sync streaming RPCs use a deadline (10s floor, always > queue window); aio client-streaming collect omits timeout because generators await empty queues. Failed send batches are counted as drops. Instance properties errors do not block keepAlive. Replacing a protocol closes the previous channel (sync close; aio await aclose on the agent loop). Timed shutdown flush so atexit cannot hang. Includes unit coverage and a multi-OAP gRPC failover E2E case.
songzhendong
force-pushed
the
feature/python-multi-backend
branch
2 times, most recently
from
August 29, 2026 14:51
d9105a4 to
d6d11de
Compare
…own, RPC margin) Skip sw_grpc instrumentation for agent collector channels via thread-local scope; move throttled reporter logs off grpc_channel so GRPC_ENABLE_FORK_SUPPORT is set before import grpc; cancel only background tasks on async shutdown; widen sync RPC deadline vs queue batch window.
Run async cleanup on the asyncio.run root after _finished is set so protocol aclose() completes before Runner teardown. Preserve the first Queue.get attempt when SW_AGENT_QUEUE_TIMEOUT=0. Add regression tests.
Wait on both the shutdown event and background tasks so failures like __command_dispatch() are retrieved and logged instead of being ignored. Retrieve outcomes from already-done tasks during shutdown cancellation.
… failures Supervisor treats early success/cancel as errors; cleanup only logs real exceptions and skips already-handled outcomes so clean shutdown stays silent. Assign the outcome marker directly to satisfy flake8 B010.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
Internal fork CI only — not for apache upstream visibility.