Skip to content

Re-enable robot state publisher service test on macOS - #56

Merged
traversaro merged 9 commits into
RoboStack:mainfrom
mini-1235:codex/reenable-robot-state-publisher-macos-test
Sep 16, 2026
Merged

traversaro merged 9 commits into
RoboStack:mainfrom
mini-1235:codex/reenable-robot-state-publisher-macos-test

Conversation

@mini-1235

@mini-1235 mini-1235 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • re-enable the robot_state_publisher parameter-service smoke test on macOS
  • bump robot_state_publisher to build 26 so CI rebuilds the package and runs the test
  • pin Cyclone DDS to the macOS lo0 interface and force multicast capability for reliable same-host discovery
  • make the tf2_ros Zenoh smoke test self-contained by starting rmw_zenohd before the test node
  • bump tf2_ros to build 26 and remove its macOS skip

What was happening

PR #54 originally ran the smoke test on macOS with:

ROS_LOCALHOST_ONLY=1 RMW_IMPLEMENTATION=rmw_cyclonedds_cpp

The macOS arm64 job started robot_state_publisher normally and received its Robot initialized message, so package loading, node creation, URDF parsing, and the initial process-output assertion all succeeded. The failure occurred afterward in the separate Python test process: its rclpy client waited 10 seconds for /robot_state_publisher/get_parameters, but Cyclone DDS never made that service visible in the client graph. The launched process then handled SIGINT and exited cleanly.

The first CI run of this PR reproduced the same sequence exactly. This isolates the failure to discovery between two healthy processes on the same GitHub macOS runner; it is not a compilation failure, a robot_state_publisher crash, or a post-shutdown exit-code problem. The pre-existing workaround skipped the service assertion on every macOS system, even though repeated local macOS runs passed.

Exact CI failure

The failure below is preserved from the macOS arm64 job in Actions run 35060140356, job 104678527551:

[INFO] [robot_state_publisher-1]: process started with pid [59882]
[robot_state_publisher-1] [INFO] [1789537048.731171000] [robot_state_publisher]: Robot initialized
test_node_output (robot_state_publisher_smoke_test_launch.TestRobotStatePublisher.test_node_output) ... ok
test_parameter_service (robot_state_publisher_smoke_test_launch.TestRobotStatePublisher.test_parameter_service) ... FAIL
======================================================================
FAIL: test_parameter_service (robot_state_publisher_smoke_test_launch.TestRobotStatePublisher.test_parameter_service)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "$SRC_DIR_run_env/etc/conda/test-files/ros2-robot-state-publisher/0/robot_state_publisher_smoke_test_launch.py", line 64, in test_parameter_service
    self.assertTrue(
    ~~~~~~~~~~~~~~~^
        client.wait_for_service(timeout_sec=10.0),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'robot_state_publisher parameter service was not ready'
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
AssertionError: False is not true : robot_state_publisher parameter service was not ready
----------------------------------------------------------------------
Ran 2 tests in 14.612s
FAILED (failures=1)
[INFO] [robot_state_publisher-1]: sending signal 'SIGINT' to process[robot_state_publisher-1]
[robot_state_publisher-1] [INFO] [1789537062.580232000] [rclcpp]: signal_handler(signum=2)
[INFO] [robot_state_publisher-1]: process has finished cleanly [pid 59882]
test_exit_codes (robot_state_publisher_smoke_test_launch.TestRobotStatePublisherPostShutdown.test_exit_codes) ... ok

This copy is kept here because GitHub Actions logs and artifacts are subject to retention and may no longer be available later.

Why the loopback configuration should help

ROS_LOCALHOST_ONLY=1 requests local-only communication, but leaves Cyclone DDS to derive the concrete network configuration. Cyclone normally ranks available interfaces and uses the operating system's reported multicast capability. That automatic choice can behave differently on the GitHub macOS runner than on a local Mac.

This PR instead supplies both processes with the same explicit Cyclone DDS configuration:

<CycloneDDS>
  <Domain>
    <General>
      <Interfaces>
        <NetworkInterface name="lo0" multicast="true" allow_multicast="true"/>
      </Interfaces>
    </General>
  </Domain>
</CycloneDDS>

Selecting lo0 prevents Cyclone from choosing another runner interface or advertising an address that the peer cannot use. Setting multicast="true" avoids relying on the interface capability reported by macOS, while allow_multicast="true" enables Cyclone's participant discovery traffic on that interface. Because the launched C++ node and the Python test process inherit the same CYCLONEDDS_URI, their discovery traffic is constrained to the same loopback transport.

This is intentionally scoped to the macOS recipe test. Other platforms retain their existing discovery configuration.

Related tf2_ros Zenoh smoke test

The tf2_ros smoke test had a separate transport setup problem. It launched static_transform_publisher with RMW_IMPLEMENTATION=rmw_zenoh_cpp and ZENOH_ROUTER_CHECK_ATTEMPTS=-1, but did not start a Zenoh router. On macOS, the process remained inside Zenoh initialization, emitted Scouting delay elapsed before start conditions are met, and never reached the expected Spinning until stopped - publishing transform message.

The updated fixture now starts rmw_zenohd, waits one second for it to initialize, and then launches static_transform_publisher with up to 10 router checks. With that setup, macOS reaches the expected node output and both the router and publisher handle SIGINT and exit cleanly. The previous Darwin skip is therefore no longer needed.

Windows originally showed the same externally visible timeout: static_transform_publisher produced Zenoh warnings but never reached its normal startup message. It is not, however, the same resolved behavior as macOS. During the investigation, starting rmw_zenohd fixed macOS, while on Windows the router process itself failed to progress through zenoh::Session::open() and never printed Started Zenoh router. That deeper Windows Zenoh initialization issue remains outside the scope of this PR; this PR does not add a Windows-specific workaround.

Validation

  • the original macOS arm64 CI failure was reproduced with the service check re-enabled
  • the launch test passes locally with Cyclone DDS explicitly configured for lo0
  • the tf2_ros Zenoh smoke test passes locally on macOS with a real router, including clean shutdown of both processes
  • pixi run generate-recipes generates the real and compatibility recipes for both changed packages at build 26
  • the macOS arm64 job in Actions run 35107492204, job 104832371269 rebuilt both packages at build 26 and passed all tests

This pull request description was AI-generated by OpenAI Codex using GPT-5.

Signed-off-by: Maurice <mauricepurnawan@gmail.com>
@mini-1235

Copy link
Copy Markdown
Contributor Author

I wasn't able to repro this locally, so opening a draft PR here to see if my agent can figure it out :(

Signed-off-by: Maurice <mauricepurnawan@gmail.com>
@mini-1235

Copy link
Copy Markdown
Contributor Author

What was happening

PR #54 originally ran the smoke test on macOS with:

ROS_LOCALHOST_ONLY=1 RMW_IMPLEMENTATION=rmw_cyclonedds_cpp

The macOS arm64 job started robot_state_publisher normally and received its Robot initialized message, so package loading, node creation, URDF parsing, and the initial process-output assertion all succeeded. The failure occurred afterward in the separate Python test process: its rclpy client waited 10 seconds for /robot_state_publisher/get_parameters, but Cyclone DDS never made that service visible in the client graph. The launched process then handled SIGINT and exited cleanly.

The first CI run of this draft reproduced the same sequence exactly. This isolates the failure to discovery between two healthy processes on the same GitHub macOS runner; it is not a compilation failure, a robot_state_publisher crash, or a post-shutdown exit-code problem. The pre-existing workaround skipped the service assertion on every macOS system, even though repeated local macOS runs passed.

Why the loopback configuration should help

ROS_LOCALHOST_ONLY=1 requests local-only communication, but leaves Cyclone DDS to derive the concrete network configuration. Cyclone normally ranks available interfaces and uses the operating system's reported multicast capability. That automatic choice can behave differently on the GitHub macOS runner than on a local Mac.

This PR instead supplies both processes with the same explicit Cyclone DDS configuration:

<CycloneDDS>
  <Domain>
    <General>
      <Interfaces>
        <NetworkInterface name="lo0" multicast="true" allow_multicast="true"/>
      </Interfaces>
    </General>
  </Domain>
</CycloneDDS>

Selecting lo0 prevents Cyclone from choosing another runner interface or advertising an address that the peer cannot use. Setting multicast="true" avoids relying on the interface capability reported by macOS, while allow_multicast="true" enables Cyclone's participant discovery traffic on that interface. Because the launched C++ node and the Python test process inherit the same CYCLONEDDS_URI, their discovery traffic is constrained to the same loopback transport.

This is intentionally scoped to the macOS recipe test. Other platforms retain their existing discovery configuration.

This seems to work @traversaro, do I need to bump robot state publisher in this case?

@mini-1235
mini-1235 marked this pull request as ready for review September 16, 2026 06:01
@traversaro

Copy link
Copy Markdown
Member

Thanks, can you add a comment to the test linking here? Otherwise the xml blob is not readable without a bit of context.

Signed-off-by: Maurice <mauricepurnawan@gmail.com>
Signed-off-by: Maurice <mauricepurnawan@gmail.com>
@mini-1235

Copy link
Copy Markdown
Contributor Author

Done, also updated the comment for next build number

@mini-1235

Copy link
Copy Markdown
Contributor Author

I am not sure why the windows test is failing

@traversaro

Copy link
Copy Markdown
Member

I am not sure why the windows test is failing

I think I merged #54 without actually waiting for the full Windows build, as I thought the test actually passed, but probably it did not actually work. Indeed, the test is also failing post-PR https://github.com/RoboStack/ros-lyrical/actions/runs/34988141755/job/104806246827 . I guess a possible action is to disable the test on Windows here, and merge.

@mini-1235

Copy link
Copy Markdown
Contributor Author

I guess a possible action is to disable the test on Windows here, and merge.

Give me a moment. I will ask my agent to take a look and see if it can figure out what is going on, even though I don't have a Windows machine to test it :)

@mini-1235

Copy link
Copy Markdown
Contributor Author

Oh actually I just noticed that tf2 ros test is also skipped for macos

Signed-off-by: Maurice <mauricepurnawan@gmail.com>
@mini-1235
mini-1235 force-pushed the codex/reenable-robot-state-publisher-macos-test branch from d32389d to edb0ebb Compare September 16, 2026 13:58
Signed-off-by: Maurice <mauricepurnawan@gmail.com>
Signed-off-by: Maurice <mauricepurnawan@gmail.com>
Signed-off-by: Maurice <mauricepurnawan@gmail.com>
@mini-1235

mini-1235 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Very strange. For the tf2 ros test, I can reproduce the windows issue on my mac (previously skipped in the CI). Applying edb0ebb fixes the issue on mac, but not on Windows

Windows originally showed the same externally visible timeout: static_transform_publisher produced Zenoh warnings but never reached its normal startup message. It is not, however, the same resolved behavior as macOS. During the investigation, starting rmw_zenohd fixed macOS, while on Windows the router process itself failed to progress through zenoh::Session::open() and never printed Started Zenoh router. That deeper Windows Zenoh initialization issue remains outside the scope of this PR; this PR does not add a Windows-specific workaround.

Signed-off-by: Maurice <mauricepurnawan@gmail.com>
@mini-1235

Copy link
Copy Markdown
Contributor Author

I am going to skip all the problematic tests on Windows first, but I will take a look again tomorrow if I can fix them

@mini-1235
mini-1235 marked this pull request as draft September 16, 2026 14:57
@traversaro

traversaro commented Sep 16, 2026

Copy link
Copy Markdown
Member

I had an agent looking locally (after reproducing the failure) and:

--- a/tests/ros-lyrical-tf2-ros/static_transform_publisher_smoke_test_launch.py
+++ b/tests/ros-lyrical-tf2-ros/static_transform_publisher_smoke_test_launch.py
@@ -20,13 +20,20 @@ def static_transform_publisher():
         additional_env={
             "RMW_IMPLEMENTATION": "rmw_zenoh_cpp",
             "RCUTILS_LOGGING_USE_STDOUT": "1",
-            "ZENOH_ROUTER_CHECK_ATTEMPTS": "-1",
+            # Flush logs immediately when launch captures stdout through a pipe.
+            "RCUTILS_LOGGING_BUFFERED_STREAM": "0",
         },
     )

 @launch_pytest.fixture
 def launch_description(static_transform_publisher):
     return LaunchDescription([
+        # Provide the router that Zenoh needs during node initialization.
+        launch_ros.actions.Node(
+            package="rmw_zenoh_cpp",
+            executable="rmw_zenohd",
+            output="screen",
+        ),
         static_transform_publisher,
     ])

worked fine (locally and starting from main).

@traversaro

Copy link
Copy Markdown
Member

I suspect he important part is RCUTILS_LOGGING_BUFFERED_STREAM=0.

@traversaro

Copy link
Copy Markdown
Member

I suspect he important part is RCUTILS_LOGGING_BUFFERED_STREAM=0.

Even if according to https://docs.ros.org/en/lyrical/Developer-Tools/Introspection-and-analysis/Logging-and-logger-configuration.html#line-buffered-console-output that is the default.

@traversaro

Copy link
Copy Markdown
Member

Anyhow, I think we can merge this as it is with the Windows skip, and improve in the future.

@mini-1235

Copy link
Copy Markdown
Contributor Author

Oh sorry, I just shut down my computer since I am heading to sleep. Feel free to push the changes to my branch if you want to try that. I also disable the robot state publisher unit test since that is failing on Windows

@mini-1235

Copy link
Copy Markdown
Contributor Author

Anyhow, I think we can merge this as it is with the Windows skip, and improve in the future.

Yeah, sure. Feel free to merge this. I may have overestimated my agent :( it seems difficult for it to fix something that can't be reproduced locally :(

@mini-1235
mini-1235 marked this pull request as ready for review September 16, 2026 15:11
@traversaro

Copy link
Copy Markdown
Member

Thanks @mini-1235 !

it seems difficult for it to fix something that can't be reproduced locally :(

Yes, delay have always being cause of instability in closed-loop controls!

@traversaro
traversaro merged commit 93fcc6c into RoboStack:main Sep 16, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants