Skip to content

fix(engine): drop the SIGALRM heartbeat cutoff that can deadlock the loop - #419

Open
MateoLostanlen wants to merge 2 commits into
developfrom
fix/heartbeat-signal-deadlock
Open

MateoLostanlen wants to merge 2 commits into
developfrom
fix/heartbeat-signal-deadlock

Conversation

@MateoLostanlen

Copy link
Copy Markdown
Member
  • On 2026-09-11 the kwartier-west engine froze for 26 h with Docker still reporting it healthy. py-spy showed the main thread stuck on a urllib3 pool lock: the heartbeat's SIGALRM raised inside the HTTP call while the lock was held, and the session finalizer then blocked on it forever.
  • The heartbeat now uses a plain 3 s requests timeout via the new Client.heartbeat(timeout=...) from feat(client): per-call timeout for heartbeat pyro-api#691, so a slow API is dropped with a warning instead of interrupting urllib3 mid-call.
  • The engine healthcheck now checks that engine.log moved in the last 70 min and kills the process when it did not, so restart: always brings a hung engine back.
  • pyroclient is pinned to the PR 691 commit; to switch back to @main and re-lock once it merges.

@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.35%. Comparing base (2465419) to head (a38b545).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #419      +/-   ##
===========================================
+ Coverage    79.11%   79.35%   +0.23%     
===========================================
  Files            6        6              
  Lines          838      833       -5     
===========================================
- Hits           663      661       -2     
+ Misses         175      172       -3     
Flag Coverage Δ
unittests 79.35% <100.00%> (+0.23%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@MateoLostanlen
MateoLostanlen marked this pull request as ready for review September 12, 2026 10:30
@MateoLostanlen
MateoLostanlen requested a review from fe51 September 12, 2026 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant