feat(runai): add Run:AI compatibility kit suite

[cparadis-nvidia]

Summary

Integrates the Run:AI compatibility kit — a prebuilt Docker image whose entrypoint runs the Run:AI E2E test suite against a Kubernetes cluster — into the step-based validation framework. The kit is the script that does the work; isvctl orchestrates it and asserts its verdict:

suites/runai.yaml → run_compatibility_kit.py (docker) → JSON contract → RunAICompatibilityCheck

What's in here

• providers/shared/run_compatibility_kit.py — stages the kubeconfig into a uid-65532 Docker volume (mirroring the kit's own Makefile flow, since the kit runtime can't read a host-owned mode-0600 bind mount), runs the kit, and parses its Allure summary + CSV into the provider-neutral contract (tests.compatibility.{passed,message,error,failed_tests}). Modes:
  • --preflight (setup): docker + kubeconfig present, image available (NGC login + pull if needed)
  • default (test): stage kubeconfig → run kit → parse results
  • --cleanup (teardown): best-effort sweep of leftover container/volume from aborted runs
• suites/runai.yaml — kubernetes-capability suite wiring the three steps and the validation. Steps are gated with requires_available_validations so the expensive docker run never happens while the check is unreleased.
• isvtest/validations/runai.py — RunAICompatibilityCheck, asserting the kit verdict and surfacing failed-test names.
• env_catalog — registers RUNAI_COMPAT_KIT_IMAGE (Run:AI section) for isvctl configure / doctor.

Design notes

• The kit image is never built here. Its build clones the private run-ai/runai-e2e-internal repo; it publishes to NVIDIA NGC without guest access. Default image: nvcr.io/nvidia/runai/runai-compatibility-kit:latest. Pulling from nvcr.io authenticates via docker login with NGC_API_KEY (key passed over stdin, never argv). A locally preloaded image (e.g. the kit Makefile's runai/certification-kit:latest) needs no credentials.
• Container exit code is not the verdict — Playwright exits non-zero on test failures, so the parsed Allure summary is the signal; pass/fail policy lives in the validation, not the script.
• Skip, not fail, when docker or a kubeconfig is missing (matches deploy_nim.py semantics; merged multi-suite runs stay green).
• The new check ships unreleased — released_tests.json untouched; run with ISVTEST_INCLUDE_UNRELEASED=1 until the release process lands it.

Testing

• 37 unit tests (scripts/tests/test_run_compatibility_kit.py, isvtest/tests/test_runai.py): parsing, verdict edge cases (broken/unknown/all-skipped), image/kubeconfig resolution precedence, NGC login (incl. key-never-in-argv), all three mode flows with docker mocked.
• make test (151 passed), make lint, make demo-test, uvx pre-commit run -a, suite wiring validator — all green.
• Full E2E against the real kit image layers: built a fake-entrypoint variant of the local runai/certification-kit:latest (same uid, same /bin/bash staging path, asserts the kubeconfig actually landed in the volume) and ran the suite end to end:

  [PASS] SETUP   : preflight_compatibility_kit: passed
  [PASS] TEST    : run_compatibility_kit: passed
    [compatibility] RunAICompatibilityCheck: PASSED - 5/7 tests passed (0 failed, 0 broken, 2 skipped)
  [PASS] TEARDOWN: cleanup_compatibility_kit: passed
  

  Zero leftover containers/volumes after the run. Also verified the no-kubeconfig skip path and the unreleased-gating path (steps skip, no docker invoked).

Open item

⚠️ The kit repo's artifact-metadata.json still says service_name: runai-certification-kit — if the publish pipeline derives the NGC image name from it, the published path will be ...runai-certification-kit, not the ...runai-compatibility-kit default used here. Needs the rename on the kit side (or a default bump here) before this leaves draft.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}