CI: Stacktraces of `pytest` and `xdist`-workers on `timeout` by ReinierMaas · Pull Request #154 · channable/opsqueue

ReinierMaas · 2026-07-24T14:20:34Z

A much better debug experience on CI. We can now see what was happening when the timeout triggered.

ReinierMaas · 2026-07-28T09:43:19Z

This works with xdist-workers:

$ OPSQUEUE_PYTEST_TIMEOUT=1 just test-integration -n1
cargo build --bin opsqueue 
   Compiling opsqueue v0.36.0 (/home/reinier/channable/opsqueue/opsqueue)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.70s
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.13 at /home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/bin/python
📡 Using build options features from pyproject.toml
Ignoring pytest: markers 'extra == "test"' don't match your environment
Ignoring pytest-xdist: markers 'extra == "test"' don't match your environment
Ignoring multiprocess: markers 'extra == "test"' don't match your environment
Ignoring pytest-timeout: markers 'extra == "test"' don't match your environment
Requirement already satisfied: cbor2 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (6.1.3)
Requirement already satisfied: opentelemetry-api in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (1.44.0)
Requirement already satisfied: opentelemetry-sdk in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (1.44.0)
Requirement already satisfied: opentelemetry-exporter-otlp in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (1.44.0)
Requirement already satisfied: typing-extensions>=4.5.0 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-api) (4.16.0)
Requirement already satisfied: opentelemetry-semantic-conventions==0.65b0 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-sdk) (0.65b0)
Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc==1.44.0 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-exporter-otlp) (1.44.0)
Requirement already satisfied: opentelemetry-exporter-otlp-proto-http==1.44.0 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-exporter-otlp) (1.44.0)
Requirement already satisfied: googleapis-common-protos~=1.57 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.44.0->opentelemetry-exporter-otlp) (1.75.0)
Requirement already satisfied: grpcio<2.0.0,>=1.66.2 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.44.0->opentelemetry-exporter-otlp) (1.82.1)
Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.44.0 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.44.0->opentelemetry-exporter-otlp) (1.44.0)
Requirement already satisfied: opentelemetry-proto==1.44.0 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-exporter-otlp-proto-grpc==1.44.0->opentelemetry-exporter-otlp) (1.44.0)
Requirement already satisfied: requests~=2.7 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-exporter-otlp-proto-http==1.44.0->opentelemetry-exporter-otlp) (2.34.2)
Requirement already satisfied: protobuf<8.0,>=5.0 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from opentelemetry-proto==1.44.0->opentelemetry-exporter-otlp-proto-grpc==1.44.0->opentelemetry-exporter-otlp) (7.35.1)
Requirement already satisfied: charset_normalizer<4,>=2 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from requests~=2.7->opentelemetry-exporter-otlp-proto-http==1.44.0->opentelemetry-exporter-otlp) (3.4.9)
Requirement already satisfied: idna<4,>=2.5 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from requests~=2.7->opentelemetry-exporter-otlp-proto-http==1.44.0->opentelemetry-exporter-otlp) (3.18)
Requirement already satisfied: urllib3<3,>=1.26 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from requests~=2.7->opentelemetry-exporter-otlp-proto-http==1.44.0->opentelemetry-exporter-otlp) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in ./.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages (from requests~=2.7->opentelemetry-exporter-otlp-proto-http==1.44.0->opentelemetry-exporter-otlp) (2026.6.17)
   Compiling opsqueue v0.36.0 (/home/reinier/channable/opsqueue/opsqueue)
   Compiling opsqueue_python v0.36.0 (/home/reinier/channable/opsqueue/libs/opsqueue_python)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.43s
📦 Built wheel for CPython 3.13 to /tmp/.tmpnVuIjl/opsqueue-0.36.0-cp313-cp313-linux_x86_64.whl
✏️ Setting installed package as editable
🛠 Installed opsqueue-0.36.0
================================================================================= test session starts ==================================================================================
platform linux -- Python 3.13.13, pytest-9.0.3, pluggy-1.6.0
rootdir: /home/reinier/channable/opsqueue/libs/opsqueue_python
configfile: pyproject.toml
plugins: xdist-3.8.0, timeout-2.4.0
timeout: 20.0s
timeout method: signal
timeout func_only: False
1 worker [103 items]   
.Thread 0x000073074dfff6c0 (most recent call first):
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 534 in read
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 567 in from_io
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 341 in run
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn

Current thread 0x0000730751958780 (most recent call first):
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/tests/conftest.py", line 103 in _handle_sigterm
  File "/nix/store/scx07dskapk27xjv1pwf11qmahfff39p-python3-3.13.13-env/lib/python3.13/threading.py", line 363 in wait
  File "/nix/store/scx07dskapk27xjv1pwf11qmahfff39p-python3-3.13.13-env/lib/python3.13/queue.py", line 210 in get
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/xdist/dsession.py", line 154 in loop_once
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/xdist/dsession.py", line 138 in pytest_runtestloop
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/main.py", line 372 in _main
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/config/__init__.py", line 199 in main
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/config/__init__.py", line 223 in console_main
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/bin/pytest", line 10 in <module>
[opsqueue pytest] Requesting stack dump from child workers: {223250}
[opsqueue pytest] ===== BEGIN CHILD STACK DUMP pid=223250 (/tmp/opsqueue-pytest-stack-223250.log) =====
Thread 0x00007557ad1ff6c0 (most recent call first):
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 534 in read
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 567 in from_io
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 341 in run
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn

Current thread 0x00007557aeee5780 (most recent call first):
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/python/opsqueue/producer.py", line 292 in blocking_stream_completed_submission_chunks
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/python/opsqueue/producer.py", line 232 in run_submission_chunks
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/python/opsqueue/producer.py", line 114 in run_submission
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/tests/test_roundtrip.py", line 58 in test_roundtrip
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/python.py", line 1720 in runtest
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/runner.py", line 179 in pytest_runtest_call
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/runner.py", line 245 in <lambda>
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/runner.py", line 353 in from_call
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/runner.py", line 244 in call_and_report
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/runner.py", line 137 in runtestprotocol
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/xdist/remote.py", line 227 in run_one_test
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/xdist/remote.py", line 206 in pytest_runtestloop
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/main.py", line 372 in _main
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/xdist/remote.py", line 427 in <module>
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 1291 in executetask
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 341 in run
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 1273 in serve
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/execnet/gateway_base.py", line 1806 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>

[opsqueue pytest] ===== END CHILD STACK DUMP pid=223250 =====
error: recipe `test-integration` failed with exit code 124

…eout`

…on `timeout`

…orkers on `timeout`

ReinierMaas · 2026-07-28T12:35:06Z

These additional outputs are super nice for debugging:

Current thread 0x0000766f0fafe780 (most recent call first):
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/python/opsqueue/producer.py", line 292 in blocking_stream_completed_submission_chunks
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/python/opsqueue/producer.py", line 232 in run_submission_chunks
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/python/opsqueue/producer.py", line 114 in run_submission
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/tests/test_roundtrip.py", line 58 in test_roundtrip

In the future we could add the handler to print stack traces to opsqueue as well but for now I only added it to the client library part.

ReinierMaas · 2026-07-28T12:37:07Z

$ OPSQUEUE_PYTEST_TIMEOUT=5 just test-integration -n1
...
[opsqueue pytest] ===== BEGIN CHILD STACK DUMP pid=296290 ('python3.13', "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/bin/python3 -s -c from multiprocess.forkserver import main; main(20, 22, ['__main__'], **{'sys_path': ['/home/reinier/channable/opsqueue/libs/opsqueue_python/tests', '/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/bin', '/nix/store/scx07dskapk27xjv1pwf11qmahfff39p-python3-3.13.13-env/lib/python313.zip', '/nix/store/scx07dskapk27xjv1pwf11qmahfff39p-python3-3.13.13-env/lib/python3.13', '/nix/store/scx07dskapk27xjv1pwf11qmahfff39p-python3-3.13.13-env/lib/python3.13/lib-dynload', '/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages', '/home/reinier/channable/opsqueue/libs/opsqueue_python/python']})") (/tmp/opsqueue-pytest-stack-296290.log) =====
Current thread 0x00007f2aee0e2780 (most recent call first):
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/python/opsqueue/consumer.py", line 139 in reserve_chunks
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/tests/test_roundtrip.py", line 99 in run_consumer
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/tests/conftest.py", line 280 in _background_main
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/multiprocess/process.py", line 108 in run
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/multiprocess/process.py", line 313 in _bootstrap
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/multiprocess/spawn.py", line 135 in _main
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/multiprocess/forkserver.py", line 323 in _serve_one
  File "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages/multiprocess/forkserver.py", line 283 in main
  File "<string>", line 1 in <module>

[opsqueue pytest] ===== END CHILD STACK DUMP pid=296290 ('python3.13', "/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/bin/python3 -s -c from multiprocess.forkserver import main; main(20, 22, ['__main__'], **{'sys_path': ['/home/reinier/channable/opsqueue/libs/opsqueue_python/tests', '/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/bin', '/nix/store/scx07dskapk27xjv1pwf11qmahfff39p-python3-3.13.13-env/lib/python313.zip', '/nix/store/scx07dskapk27xjv1pwf11qmahfff39p-python3-3.13.13-env/lib/python3.13', '/nix/store/scx07dskapk27xjv1pwf11qmahfff39p-python3-3.13.13-env/lib/python3.13/lib-dynload', '/home/reinier/channable/opsqueue/libs/opsqueue_python/.venv-928fc105951d55f0ef3cc1c72c15d87e2ac058c1/lib/python3.13/site-packages', '/home/reinier/channable/opsqueue/libs/opsqueue_python/python']})") =====
...

Here we can clearly see that we failed in reserve_chunks at the set timeout of 5 seconds (tests run for ~10-15 seconds locally so this intentionally failed somewhere during execution).

CI: Stacktraces of pytest and xdist-workers on timeout

7dd74fa

ReinierMaas self-assigned this Jul 24, 2026

ReinierMaas mentioned this pull request Jul 24, 2026

Add reservation benchmark #151

Open

ReinierMaas requested review from jerbaroo and removed request for jerbaroo July 28, 2026 09:41

ReinierMaas added 5 commits July 28, 2026 12:48

fixup! CI: Stacktraces of pytest and xdist-workers on timeout

492107f

fixup! fixup! CI: Stacktraces of pytest and xdist-workers on `tim…

62ef9ad

…eout`

fixup! fixup! fixup! CI: Stacktraces of pytest and xdist-workers …

ced1e2b

…on `timeout`

fixup! fixup! fixup! fixup! CI: Stacktraces of pytest and xdist-w…

0e4e5bc

…orkers on `timeout`

Examples: Mark processes with daemon=True

9e31f95

ReinierMaas requested a review from jerbaroo July 28, 2026 12:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CI: Stacktraces of `pytest` and `xdist`-workers on `timeout` - #154

CI: Stacktraces of `pytest` and `xdist`-workers on `timeout`#154
ReinierMaas wants to merge 6 commits into
masterfrom
reinier/ci-output-on-timeout

ReinierMaas commented Jul 24, 2026 •

edited

Loading

Uh oh!

ReinierMaas commented Jul 28, 2026

Uh oh!

ReinierMaas commented Jul 28, 2026

Uh oh!

ReinierMaas commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ReinierMaas commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ReinierMaas commented Jul 28, 2026

Uh oh!

ReinierMaas commented Jul 28, 2026

Uh oh!

ReinierMaas commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ReinierMaas commented Jul 24, 2026 •

edited

Loading