DR-008 Option 4: two-stage test-execution workflow (PR 2 of 2) by Subramanian-K812 · Pull Request #280 · eclipse-score/reference_integration

Subramanian-K812 · 2026-07-07T08:48:54Z

DR-008 Option 4: two-stage test-execution workflow (PR 2 of 2)

Closes #264. This is the second of two PRs. It is stacked on PR 1
(Subramanian-K812_resolve_override_mechanism, "resolved-dependency resolve + override
mechanism") — please review/merge that one first. Until PR 1 merges, this PR's base branch
is the PR-1 branch, so the diff shown here is only the test-execution delta.

What this PR does

Rewires the quality workflow to the DR-008 Option 4 flow, using the resolve + override
mechanism added in PR 1:

Stage 1 — integration: builds the platform, runs Feature Integration Tests, and
exports the resolved dependency set as the stage1-resolved-deps artifact
(resolved_versions.json).
prepare_matrix: derives the Stage-2 module list dynamically from
known_good.json (target_sw) — never hardcoded.
Stage 2 — per module: checks the module out at its known_good commit, overrides
its MODULE.bazel with the Stage-1 resolved set (PR 1's mechanism), and runs the
module's own unit tests + coverage as the Bazel root (//...).
aggregate: consolidates Stage 1 + Stage 2 into one quality report (and the
release-tag test-report ZIP).

Files

Workflow & runners

.github/workflows/test_and_docs.yml — two-stage restructure (+ parallel docs build)
scripts/quality_runners.py — module-context mode (--module-dir / --resolved-deps); calls ResolvedDependencies.overwrite
scripts/aggregate_quality_report.py — new, consolidated report
scripts/known_good/list_modules.py — new, dynamic Stage-2 matrix from known_good.json
scripts/integration_test.py — derive build targets from known_good.json (drop build_config.json)

Resolved pins + their config ripple (kept together so the tree is self-consistent)

known_good.json — pin bumps (baselibs, lifecycle) + per-module bazel_config + lifecycle code_root_path/extra_test_config
bazel_common/score_basic_bazel.MODULE.bazel — flatbuffers bump
bazel_common/score_modules_target_sw.MODULE.bazel — regenerated (baselibs, lifecycle)
MODULE.bazel.lock — updated lock
rust_coverage/BUILD — lifecycle query //src/... → //score/... (layout moved by the bump)
showcases/simple_lifecycle/BUILD — lifecycle bin path //src/... → //score/launch_manager
.bazelrc — coverage atomic-gcov / -no-coverage; eb-aarch64 outline-atomics link fix

Known Stage 2 integration findings (surfaced, not fixed here)

The workflow surfaces two module-owned integration failures caused by the
baselibs bump (score_logging CheckSizeValid stale uint8_t assumption;
score_persistency to_string on std::string_view).

github-actions · 2026-08-03T04:15:18Z

The created documentation from the pull request is available at: docu-html

PiotrKorkus

as the workflow has been completely reworked please create a private fork and execute it there, so we can see the proof of working in CI

pull_request_target wont allow to execute it here

Subramanian-K812 · 2026-08-17T10:48:14Z

Ran the reworked workflow end-to-end on a private fork:
https://github.com/Subramanian-K812/reference_integration-ci-verify/actions/runs/32220961400

…nsition

…job unchanged

…ule's own stale .bazelrc

OliverHeilwagen

review focus ./scripts/known_good, excluding tests.
Transitive dependency are now correctly pinned 👍 .

Patches from reference integration are locally applied but patching in Stage 2 uses single_version_override and git_override without patches, which leads to missing patches for module dependencies, which were present in Stage 1.

OliverHeilwagen · 2026-09-01T07:57:49Z

@Subramanian-K812 Thanks a lot for the quick rework 👍 . Unfortunately I am not at work till 09.09., once I am back I can continue the re-review.

…ved in eclipse-score#325)

MaximilianSoerenPollak

Some questions from my side.
But overall I think it does what was asked according to the DR.

MaximilianSoerenPollak · 2026-09-02T16:21:02Z

+            --stage1-result "${{ needs.stage1_integration.result }}" \
+            --stage2-result "${{ needs.stage2_module_validation.result }}" \
+            --stage2-dir "_stage2_reports/" \
+            >> "$GITHUB_STEP_SUMMARY"


Is there an output lenght limit for the step summary?
If there is, will this hit it or is there no risk of that?

Checked against GitHub's actual limit (1 MiB per step, upload fails with an annotation if exceeded) — a full 8-module report from the real script is ~4.3 KB, so this should not reach the cap.

MaximilianSoerenPollak · 2026-09-02T16:21:54Z

 still points at the old @score_baselibs labels, so redirect them and declare the
 score_communication dependency they now require.

+Source labels only; the module's .bazelrc has the same stale labels but ref_int no longer patches it.


But if it's stale labels should they not be removed?

We could, but that would require patching the module’s .bazelrc, which has already been ruled out for this PR.

MaximilianSoerenPollak · 2026-09-02T16:40:36Z

+    # scripts/ on sys.path, so "from known_good...." resolves under 'bazel run' as it does for
+    # "python3 scripts/...". Propagates to every consumer of this library.
+    imports = [".."],


Are you sure this does not break anything else?

Importing .. can have unexpected consequences in consumers of this py_library.

Yes agreed — better to scope or drop imports than leave it shared.

MaximilianSoerenPollak · 2026-09-02T16:41:10Z

+
+def repo_slug(repo_url: str) -> str:
+    """Derive the 'owner/name' slug actions/checkout expects from a git URL."""
+    match = re.search(r"[:/]([^/:]+/[^/:]+?)(?:\.git)?/?$", repo_url or "")


Very sure this regex does more and less than what it is suppose to do (as all regex).
But I guess it's okay for now?

This to be improved — switching to urllib.parse for the path url in known_good.json.

MaximilianSoerenPollak · 2026-09-02T16:42:10Z

+try:
+    from known_good.models.known_good import load_known_good
+    from known_good.resolved_dependencies import repo_root, workspace_path
+except ImportError:
+    if str(_HERE) not in sys.path:
+        sys.path.insert(0, str(_HERE))
+    from models.known_good import load_known_good  # noqa: E402
+    from resolved_dependencies import repo_root, workspace_path  # noqa: E402


Why is this needed again?

Agreed — same as noted below, the current try/except isn’t ideal and will be improved in a follow‑up.

MaximilianSoerenPollak · 2026-09-02T16:43:03Z

-        if module.repo in repo_commit_dict:
-            commit = repo_commit_dict[module.repo]
-
-        # Generate patches lines if bazel_patches exist
-        patches_lines = ""
-        if module.bazel_patches:
-            patches_lines = "    patches = [\n"
-            for patch in module.bazel_patches:
-                patches_lines += f'        "{patch}",\n'
-            patches_lines += "    ],\n"
-        patch_strip_line = "    patch_strip = 1,\n" if patches_lines else ""
-
-        if module.version:
-            # If version is provided, use bazel_dep with single_version_override
-            block = (
-                f'bazel_dep(name = "{module.name}")\n'
-                "single_version_override(\n"
-                f'    module_name = "{module.name}",\n'
-                f"{patch_strip_line}"
-                f"{patches_lines}"
-                f'    version = "{module.version}",\n'
-                ")\n"
-            )
-        else:
-            if not module.repo or not commit:
-                logging.warning(
-                    "Skipping module %s with missing repo or commit: repo=%s, commit=%s",
-                    module.name,
-                    module.repo,
-                    commit,
-                )
-                continue
-
-            # Validate commit hash format (7-40 hex characters)
-            if not re.match(r"^[a-fA-F0-9]{7,40}$", commit):
-                logging.warning(
-                    "Skipping module %s with invalid commit hash: %s",
-                    module.name,
-                    commit,
-                )
-                continue
-
-            # If no version, use bazel_dep with git_override
-            # Only include patch_strip if there are patches to apply
-            block = (
-                f'bazel_dep(name = "{module.name}")\n'
-                "git_override(\n"
-                f'    module_name = "{module.name}",\n'
-                f'    commit = "{commit}",\n'
-                f"{patch_strip_line}"
-                f"{patches_lines}"
-                f'    remote = "{module.repo}",\n'
-                ")\n"
-            )
-        blocks.append(block)


This whole logic not needed anymore, or moved somewhere else?

Moved, not removed — now imported from resolved_dependencies.generate_override_directive so Stage 1 and Stage 2 can't emit different git_override syntax.

MaximilianSoerenPollak · 2026-09-02T16:43:44Z

+_HERE = Path(__file__).resolve().parent
+try:
+    from known_good.module_patches import patch_relpath
+    from known_good.resolved_dependencies import (
+        INJECTED_PATCHES_PKG,
+        _collect_resolved_versions,
+        injected_override_names,
+        workspace_path,
+    )
+except ImportError:
+    if str(_HERE) not in sys.path:
+        sys.path.insert(0, str(_HERE))
+    from module_patches import patch_relpath  # noqa: E402
+    from resolved_dependencies import (  # noqa: E402
+        INJECTED_PATCHES_PKG,
+        _collect_resolved_versions,
+        injected_override_names,
+        workspace_path,
+    )


I'm sure there is a better way than to this try / except.

What are you trying to achieve with this again?

Agreed — the current try/except isn’t ideal; it’s a leftover from the same pattern used in files that are still invoked directly, which don’t have py_binary targets, so those workflows call the .py files directly instead of through bazel run.

MaximilianSoerenPollak · 2026-09-02T16:44:54Z

@@ -0,0 +1,187 @@
+#!/usr/bin/env python3


If this file runs in CI, be careful that we already had the issue (in docs-as-code) that we ran into the upper limit of a pipe buffer in the CI output (64kb) due to folding it with long prints.
Just an FYI

Thanks for the heads-up, this file's output isn't piped into anything in CI — it runs as a plain bazel run step straight to the job log, so the pipe-buffer deadlock docs-as-code hit shouldn't be an issue here.

PiotrKorkus

nested bazel invokes in ferrocene_report.sh coming from the toolchain do not carry --noworkspace_rc which makes rust coverage fail
huge differences in number of executed tests. e.g. communication has 6500 test cases under //score/... but here we run only 3800. Why do we have differences when we should execute from the score of tested repo just with common config and deps?
how about appending single line with import of common bazelrc to the one in tested module and using the config from it? Would that work or?

… shim

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch from c9462ea to fd3df76 Compare July 7, 2026 08:49

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch 2 times, most recently from d293277 to b6df34d Compare July 24, 2026 06:56

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch 2 times, most recently from e51b340 to 640dfee Compare August 3, 2026 03:26

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch 4 times, most recently from 04a7a73 to e816620 Compare August 11, 2026 10:22

PiotrKorkus requested changes Aug 14, 2026

View reviewed changes

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch 3 times, most recently from b22c037 to a992474 Compare August 21, 2026 09:58

PiotrKorkus mentioned this pull request Aug 24, 2026

Add resolved-dependency override mechanism for DR-008 #278

Merged

Subramanian-K812 added 3 commits August 24, 2026 13:04

feat: two-stage DR-008 test workflow with per-module Stage 2 execution

4624b81

ci: run new two-stage workflow alongside unchanged old job during tra…

715f615

…nsition

fix: give central-mode test runs their own exclusion list

35d83cd

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch from 587dcbb to 35d83cd Compare August 25, 2026 04:05

Subramanian-K812 marked this pull request as ready for review August 25, 2026 04:08

Subramanian-K812 requested review from AlexanderLanin, FScholPer, MaximilianSoerenPollak, antonkri, dcalavrezo-qorix, lurtz, nradakovic, opajonk and pawelrutkaq as code owners August 25, 2026 04:08

ci: split DR-008 test execution into its own workflow file, keep old …

38dd214

…job unchanged

PiotrKorkus reviewed Aug 25, 2026

View reviewed changes

Comment thread feature_integration_tests/itf/conftest.py

PiotrKorkus reviewed Aug 25, 2026

View reviewed changes

Comment thread feature_integration_tests/itf/BUILD

fix: restore itf conftest.py

05184ee

Subramanian-K812 requested a review from PiotrKorkus August 25, 2026 13:19

PiotrKorkus requested changes Aug 26, 2026

View reviewed changes

dcalavrezo-qorix reviewed Aug 27, 2026

View reviewed changes

Comment thread .github/workflows/dr8_test_execution.yml

dcalavrezo-qorix reviewed Aug 27, 2026

View reviewed changes

Comment thread .github/workflows/dr8_test_execution.yml Outdated

Subramanian-K812 added 2 commits August 28, 2026 08:18

ci(stage2): isolate score_communication under a dedicated Stage-2 rc

e24f86e

fix(stage2): isolate ferrocene_report's nested bazel calls from a mod…

743c730

…ule's own stale .bazelrc

OliverHeilwagen suggested changes Aug 28, 2026

View reviewed changes

Subramanian-K812 added 3 commits August 28, 2026 11:55

ci(stage2): drop the .bazelrc runtime rewrite for ferrocene_report

3969ba1

fix(stage2): transport dependency patches into Stage 2 injection

f3929d1

style: apply ruff formatting

e8d4ba5

antonkri mentioned this pull request Aug 31, 2026

Release v0.9 common checklist eclipse-score/score#3186

Open

23 tasks

Subramanian-K812 requested review from OliverHeilwagen, PiotrKorkus and dcalavrezo-qorix August 31, 2026 10:12

Subramanian-K812 added 2 commits September 1, 2026 08:43

fix(stage2): add score_config_management to ref_int's GCC toolchain

4fc2c62

Merge upstream/main into DR-008 branch; drop score_orchestrator (remo…

99ec4d1

…ved in eclipse-score#325)

MaximilianSoerenPollak reviewed Sep 2, 2026

View reviewed changes

PiotrKorkus reviewed Sep 2, 2026

View reviewed changes

Subramanian-K812 added 3 commits September 3, 2026 11:53

fix(stage2): isolate ferrocene_report's nested bazel calls via a PATH…

5bc1b5e

… shim

Merge upstream/main into DR-008 branch

8c69c36

Widen score_communication and score_logging code_root_path to //...

891e471

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch from 8b5251f to 891e471 Compare September 4, 2026 03:46

Uh oh!

Conversation

Subramanian-K812 commented Jul 7, 2026

DR-008 Option 4: two-stage test-execution workflow (PR 2 of 2)

What this PR does

Files

Known Stage 2 integration findings (surfaced, not fixed here)

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

PiotrKorkus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Subramanian-K812 commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OliverHeilwagen left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OliverHeilwagen commented Sep 1, 2026

Uh oh!

MaximilianSoerenPollak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Subramanian-K812 commented Aug 17, 2026 •

edited

Loading

OliverHeilwagen left a comment •

edited

Loading