ci: add Specula bug-finding CI for OpenVMM by tangruize · Pull Request #134 · microsoft/nvx

Ruize Tang (tangruize) · 2026-09-20T15:03:30Z

Move the integration from nanvix/openvmm#84 into NVX. The verification target remains OpenVMM microVM snapshot/restore.

Run on NVX published releases or manual dispatch, resolving the selected release to its exact openvmm submodule commit.
Use only runs-on: [specula], with trusted control code and manual dispatch on dev.
Retain native persistent model reuse, explicit interruption recovery, configurable verification resources, and curated report artifacts without automatically opening issues or PRs.

NVX development releases published using GITHUB_TOKEN need manual dispatch because they do not trigger another release workflow. Runner provisioning and NVX-scoped execution consent are documented in .github/specula/README.md.

No OpenVMM submodule pin, existing NVX workflow, or runner configuration is changed.

Run from microsoft/nvx on published releases or manual dispatch using the specula runner label. Resolve each NVX release to its pinned OpenVMM gitlink, retaining bounded execution, model reuse and explicit recovery. Port the integration from nanvix/openvmm#84 without changing the OpenVMM submodule pin or runner configuration.

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved security and correctness defects can bypass execution controls or make required verification unreliable.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 · 6

Open (8)

Parse or reject separate-value controlled CLI options · New
Reject version-1 authorization records for NVX execution · New
Require matching nextest tests to pass · New
Track descendant process-group cleanup after leader exit · New
Reject linked worktrees and Git alternates in retained sources · New
Require credential file ownership by the launcher user · New
Add Specula suites to pull-request validation · New
Do not publish incomplete fake backend results as baselines · New

What changed in this PR

Adds release-triggered Specula verification for the OpenVMM revision pinned by an NVX release.

Changes:

Adds release/manual orchestration with bootstrap, incremental, and resume support.
Introduces a bounded MSHV runtime and snapshot/restore harness.
Adds configuration, documentation, schemas, and regression tests.

File	Description
`.github/workflows/specula-release.yml`	Defines the Specula release workflow.
`.github/specula/tests/test_seeded_initialization.py`	Tests seeded initialization and consent.
`.github/specula/tests/test_release.py`	Tests release orchestration and recovery.
`.github/specula/tests/test_nvx_release.py`	Tests NVX release-to-gitlink resolution.
`.github/specula/tests/test_ci.py`	Tests bootstrap and CI dispatch.
`.github/specula/release.py`	Implements release selection and execution.
`.github/specula/README.md`	Documents provisioning and operation.
`.github/specula/harness/verify_cargo_tests.py`	Validates Cargo test execution.
`.github/specula/harness/tests/test_harness.py`	Tests harness cancellation and fallback behavior.
`.github/specula/harness/summarize-reuse.py`	Summarizes reuse observations.
`.github/specula/harness/run.sh`	Launches the Python harness.
`.github/specula/harness/run.py`	Runs capture and restore checks.
`.github/specula/harness/reuse-observation.rs`	Adds a device-reuse observation test.
`.github/specula/harness/result.schema.json`	Defines the harness result schema.
`.github/specula/harness/resource_check.py`	Verifies runtime resource limits.
`.github/specula/harness/README.md`	Documents harness behavior and evidence.
`.github/specula/harness/observe-reuse.sh`	Runs the reuse observation.
`.github/specula/harness/guest-init.sh`	Implements the guest snapshot workload.
`.github/specula/harness/evidence.schema.json`	Defines detailed evidence records.
`.github/specula/harness/component-tests.sh`	Runs snapshot component tests.
`.github/specula/harness/build.sh`	Builds the MSHV OpenVMM binary.
`.github/specula/guidance.md`	Defines verification scope and constraints.
`.github/specula/controller/tests/test_runtime.py`	Tests runtime policy and isolation.
`.github/specula/controller/tests_native/test_store.py`	Tests native publication storage.
`.github/specula/controller/tests_native/test_runtime_contract.py`	Tests native retry and resume contracts.
`.github/specula/controller/templates/copilot-fix.prompt.md`	Documents separately authorized follow-up.
`.github/specula/controller/store.py`	Validates and promotes native results.
`.github/specula/controller/runtime/git_guard.py`	Blocks Git publication operations.
`.github/specula/controller/runtime/entrypoint.py`	Enforces runtime policy and launches Specula.
`.github/specula/controller/runtime/deny_gh.py`	Disables the GitHub CLI.
`.github/specula/controller/runtime/copilot_wrapper.py`	Wraps model authentication and execution.
`.github/specula/controller/runtime/common.py`	Provides shared runtime utilities.
`.github/specula/controller/run.py`	Manages bounded Docker attempts.
`.github/specula/controller/Dockerfile`	Builds the pinned runtime image.
`.github/specula/controller/build-image.sh`	Builds and records the runtime image.
`.github/specula/config.json`	Configures release verification inputs.
`.github/specula/ci.py`	Coordinates bootstrap and incremental verification.
`.github/specula/authorization.example.json`	Provides a disabled consent template.
`.github/specula/.gitignore`	Excludes generated runtime files.
`.github/actionlint.yaml`	Registers the Specula runner label.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Replace the machine-specific controller and harness with a thin adapter around Specula's persistent CI workflow, provision a dedicated KVM runner, and verify the NVX-pinned OpenVMM revision directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Derive the virtualenv root from the configured binary path instead of creating a nested environment under its bin directory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Require phase artifacts to be persisted and prevent OpenVMM package restoration from creating external sysroot symlinks inside Specula's private source snapshot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ruize Tang (tangruize) · 2026-09-21T02:18:40Z

@microsoft-github-policy-service agree company="Microsoft"

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved credential exposure, command injection, authorization, and runtime-isolation risks block safe deployment.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 4 · 5

Open (9)

Reject dot path components in request and run IDs · New
Validate complete repository status before trusting source · New
Safely quote release tag input in shell commands · New
Prevent credential inheritance by analyzed source code · New
Clear report directory before each run · New
Add host-level locking for shared Specula state · New
Use SPECULA_DATA_ROOT consistently · New
Restore isolated 26 GiB runtime limits · New
Require NVX-scoped consent before token-bearing execution · New

Resolved since last review (8)

Reject version-1 authorization records for NVX execution
Parse or reject separate-value controlled CLI options
Do not publish incomplete fake backend results as baselines
Add Specula suites to pull-request validation
Require credential file ownership by the launcher user
Reject linked worktrees and Git alternates in retained sources
Track descendant process-group cleanup after leader exit
Require matching nextest tests to pass

+      SPECULA_STATE_ROOT: ${{ vars.SPECULA_STATE_ROOT || '/mnt/data/openvmm-verification' }}
+      SPECULA_TLC_MEMORY_LIMIT: ${{ vars.SPECULA_TLC_MEMORY_LIMIT || '96G' }}
+      SPECULA_TLC_WORKER_LIMIT: ${{ vars.SPECULA_TLC_WORKER_LIMIT || '16' }}


+      - name: Verify the release's pinned OpenVMM submodule
+        id: verify
+        env:
+          COPILOT_GITHUB_TOKEN: ${{ secrets.SPECULA_COPILOT_TOKEN }}
+        run: |


Pin the latest Specula main revision, use GPT-6 for the primary workflow and incremental agent, reserve GPT-5.5 for bug confirmation, and address bare caches explicitly under Git's safe repository policy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🔵 Needs a closer look

First-run verification, resume provenance, and consent enforcement remain incomplete.

Review effort: Balanced
Findings: 4 · 5

Open (9)

Prevent credential inheritance by analyzed source code
Safely quote release tag input in shell commands
Validate complete repository status before trusting source
Reject dot path components in request and run IDs
Require NVX-scoped consent before token-bearing execution
Restore isolated 26 GiB runtime limits
Use SPECULA_DATA_ROOT consistently
Add host-level locking for shared Specula state
Clear report directory before each run

Previously missed (3)

In code that hasn't changed since last review

NVX execution consent is undocumented and unenforced

.github/specula/README.md:15

The PR says this README documents NVX-scoped execution consent, but this section only explains Copilot authentication. The workflow then supplies the token and starts Specula without any documented consent artifact or validation step. Document and enforce consent scoped to this repository and verification target, or correct the stated workflow requirements before enabling it.

Resume ignores selected release revision

.github/specula/ci.py:188

Resume uses only the supplied run ID and ignores the selected release revision. An operator can therefore select release B while resuming a run created for release A; Specula resumes A's saved configuration, but publish_report() records B's revision in result.json. Validate the saved run's source revision against revision, or derive the reported revision from the saved run, before publishing the result.

Incremental request skipped after initialization

.github/specula/ci.py:353

When no current model exists, an incremental release request is replaced with a single initialization run. Since --ci-init establishes the baseline rather than verifying it, the first selected release can be reported complete without an incremental verification pass. After successful initialization, run the requested incremental phase while preserving run/report identity, or fail explicitly until initialization is completed separately.

Reject unsafe identifiers and dirty tool installations, serialize shared state, replace stale reports, quote workflow input safely, and keep Copilot credentials out of the workflow environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved correctness and credential-boundary defects can produce incomplete or mislabeled verification results.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 7 · 2

Open (9)

Avoid force-updating cached release tags · New
Pass the requested revision when resuming runs · New
Require NVX-scoped execution consent before Specula · New
Do not replace initial incremental verification with initialization · New
Isolate target execution from Copilot credentials · New
Install and validate JDK 21 and GitHub CLI · New
Replace ShellCheck-invalid compound test expression · New
Require NVX-scoped consent before token-bearing execution
Restore isolated 26 GiB runtime limits

Resolved since last review (7)

Prevent credential inheritance by analyzed source code
Safely quote release tag input in shell commands
Validate complete repository status before trusting source
Reject dot path components in request and run IDs
Use SPECULA_DATA_ROOT consistently
Add host-level locking for shared Specula state
Clear report directory before each run

Previously missed (1)

In code that hasn't changed since last review

Do not select stale runs for preflight reports

.github/specula/ci.py:302

Preflight passes an empty before set, so this condition treats every existing run directory as newly created. On a persistent runner it selects the newest old run and copies that run's summary or verdict into a report labeled preflight, publishing stale verification evidence. Skip run selection for preflight reports.

+    check_specula_source(config)
+    device = Path("/dev/kvm")
+    if not device.is_char_device() or not os.access(device, os.R_OK | os.W_OK):
+        raise CIError("runner requires readable and writable /dev/kvm")


+    if mode == "incremental" and not (ci_dir / "current").is_symlink():
+        mode = "initialize"


+        else set()
+    )
+    command = specula_command(config, mode, source, revision, run_id)
+    result = subprocess.run(command)


Reject moved cached release tags, bind resumed runs to the selected revision, prevent stale preflight evidence, and provision explicit JDK and GitHub CLI prerequisites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved security boundaries permit unrestricted agent execution and unsafe artifact collection.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 5 · 2

Open (7)

Unrestricted Copilot adapter can execute arbitrary runner operations · New
Symlinked report files can exfiltrate arbitrary host files · New
Isolate target execution from Copilot credentials
Do not replace initial incremental verification with initialization
Require NVX-scoped execution consent before Specula
Require NVX-scoped consent before token-bearing execution
Restore isolated 26 GiB runtime limits

Resolved since last review (4)

Replace ShellCheck-invalid compound test expression
Install and validate JDK 21 and GitHub CLI
Pass the requested revision when resuming runs
Avoid force-updating cached release tags

Previously missed (1)

In code that hasn't changed since last review

Virtual environment is reused after Specula commit changes

.github/specula/setup-runner.sh:49

After specula_commit changes, this reuses the existing virtual environment whenever the current imports still succeed. Because the install is editable, source code updates, but new dependency constraints and entry-point metadata are not installed; a pin update can therefore fail only in a later CI phase. Record the commit used to build the environment and recreate it whenever that marker differs from SPECULA_COMMIT, or reinstall unconditionally.

+      "agent": "copilot-cli",
+      "model": "gpt-6-astra",
+      "effort": "xhigh"


Use a minimum trusted commit and bounded semantic version range, preserve clean local descendants during provisioning, and report the exact runtime revision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Require selected run directories and curated report files to remain regular descendants of the persistent run root before copying them into uploaded artifacts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🟡 Changes recommended

Incremental reuse is broken, and credentialed execution and artifact publication lack required safeguards.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 7 · 4

Open (11)

Enforce repository and operation consent before execution · New
Prevent symlink-based artifact disclosure · New
Detect the native model directory for initialization state · New
Unrestricted Copilot adapter can execute arbitrary runner operations
Isolate target execution from Copilot credentials
Do not replace initial incremental verification with initialization
Require NVX-scoped execution consent before Specula
Preserve prerelease identifiers in semantic version checks · New
Pin Specula installation to an exact reviewed commit · New
Require NVX-scoped consent before token-bearing execution
Restore isolated 26 GiB runtime limits

Resolved since last review (1)

Symlinked report files can exfiltrate arbitrary host files

Previously missed (2)

In code that hasn't changed since last review

Publish only summary artifacts from Specula output

.github/specula/ci.py:391

This loop copies detailed ci-report.md, verdict, and resource data into the directory that the workflow uploads wholesale. The moved integration's publication boundary keeps detailed findings and evidence on the private state volume and publishes only summary.md plus the adapter-generated result.json. Limit copied Specula output to summary.md; result.json is already created below.

Require resume for unfinished native runs

.github/specula/ci.py:456

Before launching this non-resume command, the adapter never checks the native run store for an unfinished initialization or incremental run. A later request can start another expensive run instead of requiring the saved run ID, abandoning retained work and bypassing the explicit-resume recovery contract. Detect unfinished run receipts first and require the matching resume request.

+bash .github/specula/setup-runner.sh
+```
+
+The setup script installs at least the configured minimum Specula commit, Copilot CLI, GitHub CLI, JDK 21, Maven, Rust, cargo-nextest, Python environments, skills, and MCP configuration. A clean installed Specula revision is accepted when the configured minimum commit is its ancestor and its semantic version is within the configured compatibility range. This preserves reviewed local or upstream descendant commits instead of downgrading them; the exact SHA and version used by each request are recorded in `result.json`. Run `copilot login` as the dedicated runner account before starting the Actions service. Credentials are kept on the dedicated host and are not injected into the workflow environment. Specula and target commands share that dedicated account, so this integration executes only the release-selected OpenVMM commit after trusted-main ancestry validation; it is not a sandbox for untrusted pull-request code.


+        matches = (
+            sorted(selected.rglob(name), key=lambda path: len(path.parts))
+            if selected and selected.is_dir()
+            else []
+        )
+        if matches:
+            shutil.copy2(matches[0], report / name)


+    if mode == "incremental" and not (ci_dir / "current").is_symlink():
+        mode = "initialize"


+def parse_semver(value: str) -> tuple[int, int, int]:
+    match = SEMVER.fullmatch(value.strip())
+    if match is None:
+        raise CIError(f"cannot parse semantic version: {value!r}")
+    return tuple(int(component) for component in match.groups())


+if [[ ! -d "$SOURCE/.git" ]]; then
+    sudo git clone "$SPECULA_REPOSITORY" "$SOURCE"
+fi


Copilot

Copilot review overview

🟡 Changes recommended

Unresolved security, provisioning, and interruption-reporting issues can prevent safe and reliable verification.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 10 · 4

Open (14)

Symlinked runs directory permits artifact escape · New
Runner validation omits scoped execution consent · New
Rustup installer is executed without integrity verification · New
Detect the native model directory for initialization state
Prevent symlink-based artifact disclosure
Enforce repository and operation consent before execution
Unrestricted Copilot adapter can execute arbitrary runner operations
Isolate target execution from Copilot credentials
Do not replace initial incremental verification with initialization
Require NVX-scoped execution consent before Specula
Pin Specula installation to an exact reviewed commit
Preserve prerelease identifiers in semantic version checks
Require NVX-scoped consent before token-bearing execution
Restore isolated 26 GiB runtime limits

Previously missed (2)

In code that hasn't changed since last review

Interrupted runs lose result and resume metadata

.github/specula/ci.py:484

artifact_dir and the selected native run ID are published only after this blocking process exits. If Specula is interrupted or consumes the job's 720-minute timeout, this code never writes result.json or GITHUB_OUTPUT, so the always() upload steps have no report and operators lack the run ID required for the documented resume flow. Persist the run identity before launch and reserve a termination-safe step or time budget to publish the interrupted result.

Dedicated runner lacks protoc installation

.github/specula/setup-runner.sh:29

The dedicated runner does not install protoc, although the OpenVMM build scripts explicitly use /usr/bin/protoc (scripts/nvx_tools/benchmark_scripts/build-openvmm-kvm.sh:5) and the standard Linux runner installs protobuf-compiler (scripts/setup/setup-linux-runner.sh:403-408). A fresh Specula runner can therefore fail before exercising the snapshot/restore target. Add the compiler to this package list.

+    revision = resolve_nvx_tag(config, tag) if tag else args.revision
+    assert revision is not None
+    source = prepare_source(config, revision)
+    specula_identity = check_runner(config)


Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Ruize Tang <1466040111@qq.com>

Copilot

Copilot review overview

🟡 Changes recommended

New verification runs cannot start reliably, and runner compatibility and exit handling remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 9 · 4

Open (13)

Fresh run IDs are incorrectly passed as resume requests · New
Runner validation omits scoped execution consent
Detect the native model directory for initialization state
Prevent symlink-based artifact disclosure
Enforce repository and operation consent before execution
Unrestricted Copilot adapter can execute arbitrary runner operations
Isolate target execution from Copilot credentials
Do not replace initial incremental verification with initialization
Require NVX-scoped execution consent before Specula
Pin Specula installation to an exact reviewed commit
Preserve prerelease identifiers in semantic version checks
Require NVX-scoped consent before token-bearing execution
Restore isolated 26 GiB runtime limits

Resolved since last review (2)

Rustup installer is executed without integrity verification
Symlinked runs directory permits artifact escape

Previously missed (2)

In code that hasn't changed since last review

Exit code 2 is treated as success but still fails the job

.github/specula/ci.py:546

publish_report() and the emitted status both treat exit code 2 as a completed result, but returning 2 here still fails the GitHub Actions step. Normalize accepted completion codes to zero for the process exit while preserving the original Specula code in result.json.

Provisioning does not ensure Python 3.11+ is available

.github/specula/setup-runner.sh:31

Provisioning installs only the distribution's unversioned python3, but the pinned Specula requirements install github-copilot-sdk only on Python 3.11+, and line 71 imports copilot unconditionally. On hosts such as Ubuntu 22.04 that default to Python 3.10, the dependency is skipped and provisioning fails at the import check. Explicitly install and select Python 3.11+ for the virtual environment, or validate and document that prerequisite before setup begins.

+    if run_id:
+        common.append(f"--run-id={run_id}")


Copilot AI balanced review requested due to automatic review settings September 20, 2026 15:03

Copilot started reviewing on behalf of Ruize Tang (tangruize) September 20, 2026 15:04 View session

Copilot AI reviewed Sep 20, 2026

View reviewed changes

Pedro Henrique Penna (ppenna) self-assigned this Sep 20, 2026

Ubuntu and others added 2 commits September 21, 2026 01:49

ci: fix Specula virtualenv root

b9f20b1

Derive the virtualenv root from the configured binary path instead of creating a nested environment under its bin directory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings September 21, 2026 02:16

Copilot started reviewing on behalf of Ruize Tang (tangruize) September 21, 2026 02:16 View session

ci: harden Specula initialization guidance

db59fe3

Require phase artifacts to be persisted and prevent OpenVMM package restoration from creating external sysroot symlinks inside Specula's private source snapshot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>