ci: spike Moon-native task orchestration and artifact-backed cache

[kkraus14]

Important

This draft spike is based directly on NVIDIA/cuda-python:main at d6afb939e9. It supersedes the earlier implementation of this draft and does not depend on #2467.

Goal

Use Moon 2.5.1 as the repository's task graph, affected-task engine, executor, and local output cache for conditional library builds and tests. GitHub Actions remains the heterogeneous runner, credential, environment-provisioning, and durable-artifact control plane.

Relates to #299.

Constraints

• Moon uses only its system toolchain. It never installs, selects, downloads, or manages Python.
• Existing Pixi and uv manifests, lockfiles, environments, and direct commands remain supported.
• GitHub Actions continues to provision Python, CUDA toolkits, compilers, GPU runners, and provider-specific dependencies.
• OS, architecture, Python ABI/free-threaded, CUDA, package, and test granularity remain explicit.

Architecture

• One cheap gate job asks Moon for affected real tasks with moon query tasks --affected --upstream none --downstream deep. It inspects only their runner-* tags to allocate heterogeneous runner classes; it does not implement a second path classifier or package workplan.
• Allocated workers execute the same real graph with moon ci. Task commands, inputs, tags, dependencies, outputs, cache policy, and package relationships live in project moon.yml files.
• GitHub Actions owns only boundaries that Moon cannot span: operating system, architecture, Python/CUDA provisioning phases, GPU selection, credentials, Pages, and release artifacts.
• --downstream none prevents a selector from crossing runner classes. Same-environment dependencies remain in Moon's graph; cross-runner dependencies arrive as lane artifacts and declared task inputs.
• GPU tests share a Moon mutex because they mutate one installed Python environment. Independent builds run concurrently where their toolchains permit it.

Moon now parallelizes, among other work:

• bindings and core Cython test assets after one shared preparation phase;
• the previous-CTK core wheel and core test binaries;
• core and metapackage sdist builds after their common prerequisites;
• four documentation components before root assembly;
• Moon contracts, benchmark unit tests, and both API compatibility checks in one quality runner.

Moon cache through GitHub artifacts

This spike persists Moon's native local cache without treating actions/cache as a provenance boundary and without introducing an REv2 service.

• A trusted seed comes only from a successful push run of this CI workflow in the same repository, on the PR target branch, at the exact merge-base SHA.
• The baseline is 26 immutable lane bundles: two sdist lanes and 24 native OS/Python lanes.
• Each bundle carries .moon/cache/hashes, .moon/cache/outputs, and the lane's canonical .moon-out task outputs. GitHub transports the files; Moon alone interprets hashes and hydrates cached outputs.
• PR-produced bundles may feed later jobs in the same workflow run but can never seed a future run.
• Missing, expired, duplicate, malformed, or incomplete baseline bundles force the relevant lanes and start them cold.
• Rendered documentation is deliberately excluded from cross-run caching because main, PR-preview, and release render contexts differ.
• Conventional named wheel artifacts remain available for release and external consumers.

This avoids the branch-shadowing problem of actions/cache: an untrusted mirrored PR branch cannot replace the exact immutable artifact set selected from a trusted main run.

What changed

• Replaced allocation-only marker tasks with runner tags on real Moon tasks.
• Replaced package booleans in reusable workflows with affected moon ci selectors and centralized GPU matrices.
• Encoded wheel, sdist, Cython-asset, installed-test, documentation, benchmark, and local Pixi commands directly in project Moon tasks; removed the added Python dispatcher and artifact/fingerprint helper modules.
• Kept the existing core wheel merger generic and usable outside Moon; its optional directory selection and output cleanup have no repository or Moon assumptions.
• Removed the standalone pure-wheel workflow. Each native lane builds the inexpensive cuda-pathfinder wheel as a graph dependency; Linux/Python 3.12 also emits the cuda-python metapackage artifact, while tests and docs build or hydrate it on demand.
• Folded the cuda.bindings benchmarks into the bindings Moon project, leaving six workspace projects.
• Reduced each native matrix cell to externally provisioned toolchain phases while Moon owns ordering and parallelism within each phase.
• Made cached outputs isolated and lane-qualified, with direct fingerprint checks for SCM identity, Python ABI, CUDA/lane inputs, CIBW settings, resolved Cython/NumPy, and compiler/nvcc identities.
• Made the CI-only staged bindings version cohort explicit with CUDA_PYTHON_USE_STAGED_BINDINGS_VERSION=1; ordinary local metapackage builds ignore stale staged wheels.
• Kept the cuda-python Moon project named metapackage, while preserving established build-python artifact/API names at GitHub boundaries.
• Documented local Moon usage, staged toolchain phases, and the Moon-cache/GitHub-artifact boundary in CONTRIBUTING.md.

Validation

• Moon 2.5.1 project, task, action-graph, selector, and affected-query resolution
• 28 stdlib Moon task/workspace/affected-ownership contracts
• Behavioral staged-metapackage tests for local, CI, and invalid modes
• Real isolated cuda-pathfinder and cuda-python wheel builds, including a staged bindings-version cohort
• Direct benchmark smoke skip execution through Moon
• Ruff, YAML, actionlint, mypy, Cython lint, generated-file, SPDX, Lychee, and full pre-commit checks
• git diff --check origin/main...HEAD
• Independent final implementation review with no unresolved functional blocker

Known tradeoffs

• Without an RE backend, cache retention and transport remain bounded by GitHub workflow artifacts. This draft measures whether that is sufficient before adding storage infrastructure.
• The exact-base artifact boundary is trusted and immutable, but wheel/sdist tasks are not yet hermetic enough for a general cross-revision remote cache: isolated builds resolve ranged requirements and depend on runner/container compiler and repair-tool images. Production use needs pinned build constraints plus immutable toolchain identities, or those producers must stay out of persistent caching.
• Cython test-asset tasks run after GitHub changes Python/toolkit phases. Their wheels are intentionally staged inputs instead of executable Moon dependencies; local callers must stage the three current wheels first.
• Current and previous CUDA core wheels require different externally activated toolkits. They remain explicit staged Moon phases rather than one misleading local aggregate task.
• Backport branches run cold until they contain this workflow and publish a complete baseline.
• The existing mirrored-PR docs path still combines PR-controlled build code with elevated preview-deploy permissions. This spike does not expand that access; production hardening should split unprivileged rendering from a default-branch-trusted deploy stage.

Non-goals

• Removing or replacing Pixi or uv support
• Letting Moon provision or manage Python
• Moving runner, secret, CUDA/GPU provisioning, or deployment ownership out of GitHub Actions
• Introducing an REv2 service in this draft
• Redesigning release artifacts or nightly-only optional-dependency coverage

Draft exit criteria

• Cold, warm, partial, corrupt, and unavailable-baseline CI runs select the correct producers.
• No output hydrates across OS, architecture, Python ABI/free-threaded, CUDA, compiler, or build-image lanes.
• Mirrored-PR output cannot become a trusted cross-run seed.
• Live CI confirms package source/test/docs affected granularity across all runner classes.
• Direct Pixi and uv workflows remain unchanged and no Moon-created Python environment appears.
• Native/sdist isolated build dependencies and images are pinned and fingerprinted, or those outputs remain explicitly cold.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.