SolarPanelPlugin creates and casts a rendering RayQuery from PostUpdate (plugins/SolarPanelPlugin.cc, in SolarPanelPlugin::PostUpdate → UpdateLineOfSight / RayQuery::ClosestPoint). That was tolerated on Ignition Fortress, but Gazebo Harmonic runs ISystemPostUpdate systems on parallel worker threads, so once the plugins are built for Harmonic (#36) the rover's three panel instances hit the Ogre scene concurrently with each other and with the Sensors system's render thread.
Observed
Running the lunar pole exploration rover on Harmonic (gz-sim 8, ros-jazzy-gz-sim-vendor), the server corrupts its heap within seconds of the rover spawning. The failure mode varies between runs:
malloc_consolidate(): invalid chunk size
terminate called after throwing an instance of 'std::bad_alloc'
or a plain SIGSEGV / SIGBUS in the render thread. Bisecting by removing one plugin at a time from the rover's .gazebo file: only removing SolarPanelPlugin keeps the server alive; RechargeableBatteryPlugin, SensorPowerSystemPlugin and RadioisotopeThermalGeneratorPlugin do not touch rendering and are fine.
Fix
All rendering access has to happen on the render thread, i.e. in a gz::sim::events::PostRender callback (the way the Harmonic sensor systems do it). PostUpdate should only hand over the sun-to-panel ray and read back the latest line-of-sight result under a mutex. The ray query can then also be created once instead of on every step. The power formula is unchanged by that; the same structure runs on Fortress.
SolarPanelPlugincreates and casts a renderingRayQueryfromPostUpdate(plugins/SolarPanelPlugin.cc, inSolarPanelPlugin::PostUpdate→UpdateLineOfSight/RayQuery::ClosestPoint). That was tolerated on Ignition Fortress, but Gazebo Harmonic runsISystemPostUpdatesystems on parallel worker threads, so once the plugins are built for Harmonic (#36) the rover's three panel instances hit the Ogre scene concurrently with each other and with the Sensors system's render thread.Observed
Running the lunar pole exploration rover on Harmonic (gz-sim 8,
ros-jazzy-gz-sim-vendor), the server corrupts its heap within seconds of the rover spawning. The failure mode varies between runs:or a plain SIGSEGV / SIGBUS in the render thread. Bisecting by removing one plugin at a time from the rover's
.gazebofile: only removingSolarPanelPluginkeeps the server alive;RechargeableBatteryPlugin,SensorPowerSystemPluginandRadioisotopeThermalGeneratorPlugindo not touch rendering and are fine.Fix
All rendering access has to happen on the render thread, i.e. in a
gz::sim::events::PostRendercallback (the way the Harmonic sensor systems do it).PostUpdateshould only hand over the sun-to-panel ray and read back the latest line-of-sight result under a mutex. The ray query can then also be created once instead of on every step. The power formula is unchanged by that; the same structure runs on Fortress.