Throttle repeated cloud proxy rebuilds - #7536
Open
yophilav wants to merge 4 commits into
Open
Conversation
Reuse recently completed retry connection attempts per device so queued operations cannot trigger a sequential rebuild stampede when replacements immediately become inactive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0b428cdd-2248-4f74-87b1-cd02dc9ad50a
Coalesce failed per-device rebuild attempts, fence concurrent removals, and preserve token handoff without exposing stale connections to normal lookup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0b428cdd-2248-4f74-87b1-cd02dc9ad50a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0b428cdd-2248-4f74-87b1-cd02dc9ad50a
Make retry timing and generation invariants explicit, avoid starting asynchronous creation under the state lock, and replace fixed-delay token tests with deterministic synchronization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0b428cdd-2248-4f74-87b1-cd02dc9ad50a
There was a problem hiding this comment.
Pull request overview
This PR updates Edge Hub’s cloud-connection management to prevent “stampede” rebuilds of per-device cloud proxies during upstream outages by introducing per-device creation throttling and stronger invariants around creation/removal lifecycles. It also hardens token-refresh behavior so discarded connections don’t keep accepting or delivering tokens, and adds deterministic unit tests around these new behaviors.
Changes:
- Add per-device cooldown/throttle behavior for repeated failed or immediately-inactive cloud connection creation in
ConnectionManager. - Introduce preserved-connection semantics for token-handoff plus serialized cloud-connection removal with generation invalidation to discard stale create/update results.
- Add cancelable token update lifecycle in
ClientTokenCloudConnection, plus new unit tests to validate cancellation and throttling behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/ConnectionManager.cs | Implements per-device cloud-connection creation throttling, preserves token-handoff state, and serializes/removes with generation-based invalidation. |
| edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/cloud/IClientTokenCloudConnection.cs | Extends the cloud-connection token interface to expose pending-token state and allow cancellation. |
| edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ClientTokenCloudConnection.cs | Implements cancelable token waiter lifecycle and suppresses token-status callbacks after cancellation. |
| edge-hub/core/test/Microsoft.Azure.Devices.Edge.Hub.Core.Test/ConnectionManagerTest.cs | Adds tests validating throttling and removal/invalidated in-flight connection behavior. |
| edge-hub/core/test/Microsoft.Azure.Devices.Edge.Hub.Core.Test/RetryingCloudProxyTest.cs | Updates tests to inject a monotonic clock and disable retry delay for determinism. |
| edge-hub/core/test/Microsoft.Azure.Devices.Edge.Hub.CloudProxy.Test/ClientTokenCloudConnectionTest.cs | Adds tests ensuring cancellation rejects token requests and suppresses callbacks during retry. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+9
to
13
| bool HasPendingTokenUpdate { get; } | ||
|
|
||
| void CancelTokenUpdate(); | ||
|
|
||
| Task<ICloudProxy> UpdateTokenAsync(ITokenCredentials tokenCredentials); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TimeSpan, avoiding wall-clock and raw tick-unit coupling.TaskCompletionSourcetoken waiter lifecycle, stale callback suppression, and cancel-before-close cleanup for discarded connections.Fixes #7535
Testing
ConnectionManagerTestandRetryingCloudProxyTest: 34 passedClientTokenCloudConnectionTest: 9 passedCheckInBuild/ StyleCop: passed