diff --git a/skills/README.md b/skills/README.md index 8822b2cac..98452bf89 100644 --- a/skills/README.md +++ b/skills/README.md @@ -13,6 +13,7 @@ agent (or person) can simply read the relevant `SKILL.md` before acting. |-------|----------| | [deploy-slurm-cluster](deploy-slurm-cluster/SKILL.md) | Deploying or rebuilding a Slurm GPU cluster. | | [deploy-k8s-gpu-cluster](deploy-k8s-gpu-cluster/SKILL.md) | Deploying or rebuilding a Kubernetes GPU cluster. | +| [deploy-airgapped](deploy-airgapped/SKILL.md) | Preparing mirrors and deploying Slurm or Kubernetes without Internet access. | | [validate-gpu-cluster](validate-gpu-cluster/SKILL.md) | Health checks and post-deploy verification. | | [diagnose-driver-install](diagnose-driver-install/SKILL.md) | NVIDIA driver failures, `nvidia-smi` errors, GPU pods crash-looping. | diff --git a/skills/deploy-airgapped/SKILL.md b/skills/deploy-airgapped/SKILL.md new file mode 100644 index 000000000..96fd87263 --- /dev/null +++ b/skills/deploy-airgapped/SKILL.md @@ -0,0 +1,999 @@ +--- +name: deploy-airgapped +description: Prepare mirrors and transfer artifacts, configure DeepOps, deploy Slurm or Kubernetes GPU clusters without Internet access, and validate them with machine-readable gates. Use for disconnected, restricted-egress, offline, or air-gapped DeepOps installations and for diagnosing missing package, file, chart, or container artifacts. +--- + +# Deploy DeepOps in an air-gapped environment + +## Preconditions and boundaries + +- Use an Internet-connected staging machine and a separate provisioning machine + that can reach every target and every internal mirror. Do not connect an + isolated network to the Internet for this procedure. +- Fix one DeepOps commit, target OS release, CPU architecture, cluster type, + inventory, and enabled feature set before mirroring. Repository and image + requirements change when any of these inputs change. +- Provide complete distribution package mirrors plus internal HTTP package/file + service and container registry endpoints. Use trusted TLS where possible; if + the registry is intentionally insecure, configure it explicitly. +- Bootstrap the isolated provisioning machine with the OS packages and Python + environment required by `scripts/setup.sh` before closing the boundary, or + provide approved internal OS and Python package indexes. Copying the source + tree alone does not install Ansible. +- Reserve enough storage for repository metadata, packages, container images, + charts, the initialized DeepOps checkout, and two copies during transfer. +- Confirm that every target is fully manageable and has no active users or + workloads. Driver installation can reboot hosts. +- Run DeepOps commands from the repository root. Keep secrets out of archives + and store site configuration in `config/`, never `config.example/`. + +DeepOps does not provide supported mirror-building automation. Do **not** run +`scripts/airgap/build_offline_cache.sh`: it calls the removed +`playbooks/airgap/build-offline-cache.yml` and cannot complete in this tree. +Build and verify the mirrors explicitly as described below. + +## 1. Freeze the deployment inputs + +1. Record the DeepOps commit and initialize every submodule on the connected + staging machine: + + ```bash + git rev-parse HEAD + git submodule update --init --recursive + ./scripts/setup.sh + ``` + +2. Copy `config.example/` to `config/`. Define the real hosts in + `config/inventory` before collecting artifacts so the selected cluster path + and optional components are known: + + - Slurm: controller/login hosts in `[slurm-master]`, compute hosts in + `[slurm-node]`; a single host may be in both groups. + - Kubernetes: control-plane hosts in `[kube_control_plane]` and `[etcd]`, + workers in `[kube_node]`; a single host may be in all three groups. + +3. Review the resolved inventory. Stop if it contains any unintended host: + + ```bash + ansible-inventory -i config/inventory --list + ``` + +4. Record the target OS release/architecture and all enabled roles. Mirror + every dependency of that exact profile. In particular, account for: + + - distribution, Docker CE, NVIDIA CUDA/driver, NVIDIA Container Toolkit, + and EPEL repositories where applicable; + - direct-download archives referenced by enabled role defaults; + - every container image and Helm chart used by the chosen Slurm or + Kubernetes path; + - the initialized Kubespray submodule, Galaxy roles/collections, and Python + packages needed on the provisioning machine. + +## 2. Build package mirrors on the connected side + +### Ubuntu/APT + +Install `apt-mirror`, set `base_path` in `/etc/apt/mirror.list` (for example, +`/var/repos`), and add the suites for the target release. For Ubuntu 24.04 +(`noble`), the current DeepOps documentation uses these sources: + +```text +deb http://archive.ubuntu.com/ubuntu noble main restricted universe multiverse +deb http://archive.ubuntu.com/ubuntu noble-security main restricted universe multiverse +deb http://archive.ubuntu.com/ubuntu noble-updates main restricted universe multiverse +deb http://archive.ubuntu.com/ubuntu noble-backports main restricted universe multiverse +deb https://download.docker.com/linux/ubuntu noble stable +deb https://nvidia.github.io/libnvidia-container/stable/deb/amd64 / +deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64 / +``` + +Add DGX OS or MAAS repositories only when the selected profile uses them. Use +the matching suite and architecture instead of `noble`/`amd64` for another +target. Download exactly the configured set: + +```bash +sudo mkdir -p /var/repos +sudo apt-mirror +``` + +Download the signing material that DeepOps fetches directly, plus the RPM +bootstrap files when building an Enterprise Linux mirror. Keep these outside +`mirror/` so their archive paths are deterministic: + +```bash +sudo mkdir -p /var/repos/keys /var/repos/bootstrap +sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \ + -o /var/repos/keys/docker-ubuntu.asc +sudo curl -fsSL https://download.docker.com/linux/centos/gpg \ + -o /var/repos/keys/docker-rpm.asc +sudo curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ + -o /var/repos/keys/libnvidia-container.asc +sudo curl -fsSL \ + https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub \ + -o /var/repos/keys/cuda-ubuntu2404-3bf863cc.pub +sudo curl -fsSL \ + https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/D42D0685.pub \ + -o /var/repos/keys/cuda-rhel9-D42D0685.pub +sudo curl -fsSL \ + https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-9 \ + -o /var/repos/keys/RPM-GPG-KEY-EPEL-9 +sudo curl -fsSL \ + https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \ + -o /var/repos/bootstrap/epel-release-latest-9.noarch.rpm +gpg --show-keys --with-fingerprint /var/repos/keys/* +``` + +Replace `ubuntu2404`, `rhel9`, and `9` together for another supported target. +Verify the downloaded fingerprints against the vendor's published values under +the site's package-trust process before transfer; do not turn off GPG checking. + +### Enterprise Linux/RPM + +Configure the target distribution repositories plus EPEL, Docker CE, CUDA, +`libnvidia-container`, and `nvidia-container-toolkit`. For EL9, download the +checked-in documentation's complete additional-repository set into the +top-level repo-ID directories created by `reposync`: + +```bash +for repo_id in epel docker-ce-stable cuda-rhel9-x86_64 \ + libnvidia-container nvidia-container-toolkit; do + sudo reposync -l --repoid="${repo_id}" --downloadcomps \ + --download-metadata --download_path=/var/repos + sudo createrepo "/var/repos/${repo_id}" +done +``` + +Use `cuda-rhel8-x86_64` instead on EL8. Compare the final IDs and package +contents with `/etc/yum.repos.d/` and the enabled workload on a connected +reference host; distribution repositories remain a separate prerequisite. + +## 3. Mirror direct files, charts, and images + +1. Download the direct files selected by the current default Slurm profile. + These names and versions come from the checked-in role defaults: + + ```bash + sudo install -d -m 0755 /var/repos/downloads + while read -r url name; do + sudo curl -fL "${url}" -o "/var/repos/downloads/${name}" + done <<'EOF' + https://download.schedmd.com/slurm/slurm-26.05.1.tar.bz2 slurm-26.05.1.tar.bz2 + https://download.open-mpi.org/release/hwloc/v2.5/hwloc-2.5.0.tar.gz hwloc-2.5.0.tar.gz + https://github.com/openpmix/openpmix/releases/download/v3.2.3/pmix-3.2.3.tar.bz2 pmix-3.2.3.tar.bz2 + https://github.com/mej/nhc/releases/download/1.4.3/lbnl-nhc-1.4.3.tar.xz lbnl-nhc-1.4.3.tar.xz + https://github.com/NVIDIA/pyxis/archive/v0.11.1.tar.gz pyxis-0.11.1.tar.gz + https://developer.download.nvidia.com/hpc-sdk/23.7/nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz + https://get.helm.sh/helm-v3.17.1-linux-amd64.tar.gz helm-v3.17.1-linux-amd64.tar.gz + EOF + for name in slurm-26.05.1.tar.bz2 hwloc-2.5.0.tar.gz \ + pmix-3.2.3.tar.bz2 lbnl-nhc-1.4.3.tar.xz pyxis-0.11.1.tar.gz \ + nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz \ + helm-v3.17.1-linux-amd64.tar.gz; do + test -s "/var/repos/downloads/${name}" + done + ``` + + Enroot is also enabled by default, but its package filenames are owned by + the pinned `nvidia.enroot` Galaxy role rather than this repository. After + `./scripts/setup.sh`, use that installed role's real package list to + download the target-OS packages and generate the matching Ansible override. + Choose `enroot_deb_packages` for Ubuntu or `enroot_rpm_packages` for + Enterprise Linux; repeat for both when the inventory mixes OS families: + + ```bash + test -s roles/galaxy/nvidia.enroot/defaults/main.yml + cat >/tmp/deepops-mirror-enroot.yml <<'EOF' + --- + - hosts: localhost + connection: local + gather_facts: true + become: true + vars_files: + - "{{ deepops_root }}/roles/galaxy/nvidia.enroot/defaults/main.yml" + tasks: + - name: Download the pinned Enroot packages + get_url: + url: "{{ item }}" + dest: "/var/repos/downloads/{{ item | basename }}" + mode: "0644" + loop: "{{ lookup('vars', enroot_package_list_var) }}" + - name: Write the matching offline override + copy: + dest: "/var/repos/downloads/{{ enroot_package_list_var }}-offline.yml" + mode: "0644" + content: | + {{ enroot_package_list_var }}: + {% for package_url in lookup('vars', enroot_package_list_var) %} + - "{{ offline_http_base }}/{{ package_url | basename }}" + {% endfor %} + EOF + ansible-playbook -i localhost, /tmp/deepops-mirror-enroot.yml \ + -e "deepops_root=$(pwd)" \ + -e enroot_package_list_var=enroot_deb_packages \ + -e offline_http_base=http://package-server/downloads + ``` + + Replace `package-server` before running the generator. Verify every file is + nonempty. Use these exact overrides with the published paths: + + ```yaml + slurm_src_url: "http://package-server/downloads/slurm-26.05.1.tar.bz2" + hwloc_src_url: "http://package-server/downloads/hwloc-2.5.0.tar.gz" + pmix_src_url: "http://package-server/downloads/pmix-3.2.3.tar.bz2" + nhc_src_url: "http://package-server/downloads/lbnl-nhc-1.4.3.tar.xz" + slurm_pyxis_tarball_url: "http://package-server/downloads/pyxis-0.11.1.tar.gz" + hpcsdk_download_url: "http://package-server/downloads/nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz" + ``` + + Add the generated Enroot YAML list to `config/group_vars/all.yml`. On + Enterprise Linux, also override `epel_package` and `epel_key_url`; several + default roles import the EPEL key directly, so mirroring packages alone is + insufficient. + + DCGM needs no separate file mirror: the `nvidia_dcgm` role installs + `dcgm_pkg_name` (`datacenter-gpu-manager`) from the CUDA package + repository, so it is covered by the CUDA repository mirror below — verify + the DCGM package is present in the mirrored CUDA repo content. + + The default Slurm configuration also enables CUDA installation, whose + Ubuntu and Enterprise Linux tasks fetch the CUDA repository and its + signing material from `developer.download.nvidia.com`. Mirroring the CUDA + packages alone does not redirect these tasks. On Ubuntu, the role uses + `nvidia_driver_ubuntu_cuda_keyring_url` only to install the keyring package; + it does **not** use `nvidia_driver_ubuntu_cuda_repo_baseurl` to write an APT + source. The keyring package writes an upstream source that must be replaced + on every target as shown in step 5. On Enterprise Linux, override both + repository variables after mirroring the metadata and GPG key: + + - `nvidia_driver_ubuntu_cuda_keyring_url` (Ubuntu, plus the target-side APT + source replacement in step 5) + - `nvidia_driver_rhel_cuda_repo_baseurl` and + `nvidia_driver_rhel_cuda_repo_gpgkey` (Enterprise Linux) + + Either mirror and override every enabled dependency or explicitly disable + the optional component. Do not silently fall back to its public URL. + +2. Mirror the Helm repositories/charts required by the selected Kubernetes + profile. The default tree references the stable Helm repository, GPU + Operator `v26.3.3`, and, when enabled, NFS subdir external provisioner + `4.0.18`. Set `gpu_operator_helm_repo` and + `k8s_nfs_client_helm_repo` to internal chart repositories. + + Archive the chart versions selected by current DeepOps defaults on the + connected side: + + ```bash + mkdir -p /tmp/charts + helm repo add nvidia https://helm.ngc.nvidia.com/nvidia + helm pull nvidia/gpu-operator --version v26.3.3 --destination /tmp/charts + helm repo add nfs-subdir-external-provisioner \ + https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner + helm pull nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \ + --version 4.0.18 --destination /tmp/charts + helm repo index /tmp/charts + ``` + + Omit the NFS chart only when `k8s_nfs_client_provisioner: false`. Publish + `/tmp/charts` from an internal HTTP service and set the two repository + variables to that URL. + +3. Generate and collect the exact Kubespray file and image set from the pinned + submodule. `generate_list.yml` runs only against `localhost`; passing + `-i config/inventory` alone does **not** apply variables from DeepOps' + `k8s_cluster` group. Export the effective variables for one representative + Kubernetes host, review the selected versions, and pass that JSON as + extra-vars so the localhost play uses the deployment's resolved overrides. + Kubernetes download versions must be cluster-wide; stop if control-plane and + worker hosts resolve different artifact-selection variables. Then render the + enabled DeepOps charts with the same values used by their roles and append + their images to Kubespray's list before creating the image archive: + + ```bash + task_deepops_root=$(pwd) + task_offline_dir="${task_deepops_root}/submodules/kubespray/contrib/offline" + task_kube_host=kube01 # replace with one real kube_control_plane or kube_node host + ansible-inventory -i config/inventory --host "${task_kube_host}" \ + > /tmp/kubespray-effective-vars.json + python3 - /tmp/kubespray-effective-vars.json <<'PY' + import json + import sys + + variables = json.load(open(sys.argv[1], encoding="utf-8")) + for name in ("kube_version", "container_manager", "etcd_deployment_type"): + print(f"{name}={variables.get(name, '')}") + PY + cd "${task_offline_dir}" + ./generate_list.sh -i "${task_deepops_root}/config/inventory" \ + -e "@/tmp/kubespray-effective-vars.json" + test -s temp/files.list + test -s temp/images.list + NO_HTTP_SERVER=1 ./manage-offline-files.sh + + helm template nvidia-gpu-operator \ + /tmp/charts/gpu-operator-v26.3.3.tgz \ + --namespace gpu-operator \ + --set driver.version=580.126.20 \ + --set mig.strategy=mixed \ + --set driver.enabled=false \ + --set toolkit.enabled=true \ + --set dcgm.enabled=false \ + --set migManager.enabled=true \ + > /tmp/gpu-operator-rendered.yml + helm template nfs-subdir-external-provisioner \ + /tmp/charts/nfs-subdir-external-provisioner-4.0.18.tgz \ + > /tmp/nfs-provisioner-rendered.yml + python3 - /tmp/gpu-operator-rendered.yml \ + /tmp/nfs-provisioner-rendered.yml <<'PY' >> temp/images.list + import sys + import yaml + + images = set() + + def walk(value): + if isinstance(value, dict): + repository = value.get("repository") + image = value.get("image") + version = value.get("version", value.get("tag")) + if isinstance(repository, str) and isinstance(image, str) and version: + reference = f"{repository.rstrip('/')}/{image.lstrip('/')}" + if "@" not in reference and ":" not in reference.rsplit("/", 1)[-1]: + separator = "@" if str(version).startswith("sha256:") else ":" + reference = f"{reference}{separator}{version}" + images.add(reference) + if isinstance(image, str): + tail = image.rsplit("/", 1)[-1] + if "@" in image or ":" in tail: + images.add(image) + for child in value.values(): + walk(child) + elif isinstance(value, list): + for child in value: + walk(child) + + for manifest_path in sys.argv[1:]: + with open(manifest_path, encoding="utf-8") as stream: + for document in yaml.safe_load_all(stream): + walk(document) + + if not images: + raise SystemExit("no chart images found") + print("\n".join(sorted(images))) + PY + sort -u -o temp/images.list temp/images.list + test -z "$(grep -Ev \ + '^[^[:space:]]+(:[^/[:space:]]+|@sha256:[0-9a-f]+)$' \ + temp/images.list)" + + PRIVATE_REGISTRY=nvcr.io IMAGES_FROM_FILE="${task_offline_dir}/temp/images.list" \ + ./manage-offline-container-images.sh create + install -d /tmp/kubespray-offline + cp offline-files.tar.gz container-images.tar.gz temp/files.list \ + temp/images.list /tmp/kubespray-offline/ + cd "${task_deepops_root}" + ``` + + The `PRIVATE_REGISTRY=nvcr.io` setting makes the checked-in image manager + strip `nvcr.io/` just as it strips the Kubespray source registries, so an + internal registry mirror can serve the paths that the GPU Operator chart + requests. If NFS is disabled, omit its `helm template` command. If site + overrides change any GPU Operator flag, render with those values instead. + The inventory path is still required by the generator, but the explicit + effective-variable JSON is what makes group/host version overrides visible + to its localhost play. Treat a missing representative host, an empty list, a + failed pull, or an unparseable rendered image as a collection failure; do not + continue with a partial archive. + +4. Pull and archive the CUDA validator plus the images used by enabled Slurm + monitoring/registry roles. For the current validator and registry images: + + ```bash + mkdir -p /tmp/images + docker pull nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 + docker save -o /tmp/images/nvidia-cuda-12.4.1-base-ubuntu22.04.tar \ + nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 + docker pull registry:3.1.1 + docker save -o /tmp/images/registry-3.1.1.tar registry:3.1.1 + ``` + + When Slurm monitoring stays enabled, also mirror the images selected by + `prometheus_container`, `grafana_container`, `alertmanager_container`, + `node_exporter_container`, and `nvidia_dcgm_container`, then override those + variables with their internal registry paths. The Slurm exporter is a + special case: its default `slurm_exporter_build_image: true` builds from + `golang:1.24` and `ubuntu:24.04`, and the runtime stage runs `apt-get`. + Build it while connected, archive it with the other images, and use the + prebuilt image offline: + + ```bash + for image in prom/prometheus:v3.13.0 grafana/grafana:13.1.0 \ + prom/alertmanager:v0.33.0 quay.io/prometheus/node-exporter:v1.11.1 \ + nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless; do + docker pull "${image}" + done + docker save -o /tmp/images/slurm-monitoring.tar \ + prom/prometheus:v3.13.0 grafana/grafana:13.1.0 \ + prom/alertmanager:v0.33.0 quay.io/prometheus/node-exporter:v1.11.1 \ + nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless + docker build -t deepops/prometheus-slurm-exporter:2.0.0 \ + roles/prometheus-slurm-exporter/files/docker + docker save -o /tmp/images/prometheus-slurm-exporter-2.0.0.tar \ + deepops/prometheus-slurm-exporter:2.0.0 + ``` + + This connected-side build obtains the two base images and runtime APT + packages. Do not leave the default local build enabled in the isolated + environment unless an approved internal build setup supplies both base + images and APT sources. + +5. Archive the initialized checkout separately from site secrets. **Keep + `.git` in the archive**: `playbooks/k8s-cluster.yml` unconditionally runs + `git submodule update --init` from the repository root, so an extracted + tree without Git metadata fails before Kubespray starts. With submodules + already initialized at the pinned commits, that task is an offline no-op. + Run this from the checkout's parent directory after `setup.sh` has + populated its Galaxy dependencies and + `git submodule update --init --recursive` has completed: + + ```bash + git -C deepops submodule status --recursive # every line must start with a space (initialized, pinned) + tar --exclude='deepops/config' \ + -czf /tmp/deepops-source.tar.gz deepops + ``` + + The archive is larger with Git metadata included; that is the price of the + Kubernetes path working. Excluding `deepops/config` still keeps site + secrets out of the transfer artifact. + +6. Package repositories and images for approved removable-media transfer and + create checksums. Substitute a protected, user-owned staging directory for + `/path/to/staging`; create the ISO files without elevated privileges: + + ```bash + genisoimage -o /path/to/staging/packages.iso /var/repos + genisoimage -o /path/to/staging/images.iso /tmp/images + genisoimage -o /path/to/staging/charts.iso /tmp/charts + genisoimage -o /path/to/staging/kubespray-offline.iso /tmp/kubespray-offline + genisoimage -o /path/to/staging/keys.iso /var/repos/keys + cp /tmp/deepops-source.tar.gz /path/to/staging/ + cd /path/to/staging + sha256sum packages.iso images.iso charts.iso kubespray-offline.iso keys.iso \ + deepops-source.tar.gz > SHA256SUMS + ``` + +Transfer the ISO files, checkout archive, and checksum file through the site's +approved boundary process. Verify the checksums again +inside the isolated network before importing anything: + +```bash +cd /path/to/import +sha256sum -c SHA256SUMS +``` + +## 4. Import artifacts on the isolated side + +1. Mount the verified transfer images and copy their contents to persistent or + working directories: + + ```bash + sudo mkdir -p /mnt/deepops-packages /mnt/deepops-images \ + /mnt/deepops-charts /mnt/deepops-kubespray /mnt/deepops-keys + sudo mount -o loop /path/to/import/packages.iso /mnt/deepops-packages + sudo mount -o loop /path/to/import/images.iso /mnt/deepops-images + sudo mount -o loop /path/to/import/charts.iso /mnt/deepops-charts + sudo mount -o loop /path/to/import/kubespray-offline.iso /mnt/deepops-kubespray + sudo mount -o loop /path/to/import/keys.iso /mnt/deepops-keys + sudo mkdir -p /var/repos /var/www/html/charts + mkdir -p /tmp/images + sudo cp -a /mnt/deepops-packages/. /var/repos/ + cp -a /mnt/deepops-images/. /tmp/images/ + sudo cp -a /mnt/deepops-charts/. /var/www/html/charts/ + mkdir -p /tmp/kubespray-offline + cp -a /mnt/deepops-kubespray/. /tmp/kubespray-offline/ + ``` + +2. Publish signing keys first, and fail if any expected file is missing. This + EL9/Ubuntu 24.04 example corresponds exactly to the connected-side commands: + + ```bash + sudo install -d /var/www/html/keys + for key in docker-ubuntu.asc docker-rpm.asc libnvidia-container.asc \ + cuda-ubuntu2404-3bf863cc.pub cuda-rhel9-D42D0685.pub \ + RPM-GPG-KEY-EPEL-9; do + test -s "/mnt/deepops-keys/${key}" + sudo install -m 0644 "/mnt/deepops-keys/${key}" "/var/www/html/keys/${key}" + done + gpg --show-keys --with-fingerprint /var/www/html/keys/* + sha256sum /var/www/html/keys/* + ``` + + For APT, preserve `apt-mirror`'s hostname and path layout verbatim: + + ```bash + sudo install -d /var/www/html/repos + test -d /var/repos/mirror/archive.ubuntu.com/ubuntu/dists/noble + test -d /var/repos/mirror/download.docker.com/linux/ubuntu/dists/noble + test -d /var/repos/mirror/nvidia.github.io/libnvidia-container/stable/deb/amd64 + test -d /var/repos/mirror/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64 + sudo cp -a /var/repos/mirror/. /var/www/html/repos/ + ``` + + Publish the direct Slurm/Enroot files under the paths used by the generated + overrides: + + ```bash + test -d /var/repos/downloads + sudo install -d /var/www/html/downloads + sudo cp -a /var/repos/downloads/. /var/www/html/downloads/ + test -z "$(find /var/www/html/downloads -maxdepth 1 -type f -size 0 -print -quit)" + ``` + + For RPM, publish the exact top-level repo-ID directories, EPEL bootstrap + RPM, and a rewritten Container Toolkit repo file whose base URLs are + internal: + + ```bash + for repo_id in epel docker-ce-stable cuda-rhel9-x86_64 \ + libnvidia-container nvidia-container-toolkit; do + test -d "/var/repos/${repo_id}/repodata" + sudo cp -a "/var/repos/${repo_id}" /var/www/html/repos/ + done + test -s /var/repos/bootstrap/epel-release-latest-9.noarch.rpm + sudo install -m 0644 /var/repos/bootstrap/epel-release-latest-9.noarch.rpm \ + /var/www/html/repos/epel-release-latest-9.noarch.rpm + sudo tee /var/www/html/repos/nvidia-container-toolkit.repo >/dev/null <<'EOF' + [libnvidia-container] + name=libnvidia-container + baseurl=http://package-server/repos/libnvidia-container + enabled=1 + gpgcheck=1 + gpgkey=http://package-server/keys/libnvidia-container.asc + + [nvidia-container-toolkit] + name=nvidia-container-toolkit + baseurl=http://package-server/repos/nvidia-container-toolkit + enabled=1 + gpgcheck=1 + gpgkey=http://package-server/keys/libnvidia-container.asc + EOF + ``` + + Replace `package-server` with the real internal DNS name in both the repo + file and step 5. Verify every configured file URL with `curl -fsS` and every + APT/RPM metadata path with the target package manager before deployment. + Preserve GPG verification. + +3. Load the registry image and start the isolated registry if the site does + not already provide one. Run this fallback on the registry host itself. It + is plain HTTP, so the imports below push through Docker's IPv4 loopback + exception (`127.0.0.1:5000`); cluster nodes use the reachable + `registry-host:5000` name configured later. Do not push to + `registry-host:5000` from the importer unless that Docker daemon has been + separately and deliberately configured for the insecure endpoint: + + ```bash + docker load -i /tmp/images/registry-3.1.1.tar + docker volume create registry-images + docker run -d -p 5000:5000 --restart=always --name registry \ + -v registry-images:/var/lib/registry registry:3.1.1 + ``` + +4. Import, retag, and push every archived image on that registry host. Preserve + the complete repository path expected by the consuming chart or role. The + hostname used for the push is not part of the stored repository path, so a + cluster pull of `registry-host:5000/nvidia/cuda:...` resolves the image + pushed as `127.0.0.1:5000/nvidia/cuda:...`: + + ```bash + docker load -i /tmp/images/nvidia-cuda-12.4.1-base-ubuntu22.04.tar + docker tag nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 \ + 127.0.0.1:5000/nvidia/cuda:12.4.1-base-ubuntu22.04 + docker push 127.0.0.1:5000/nvidia/cuda:12.4.1-base-ubuntu22.04 + docker load -i /tmp/images/prometheus-slurm-exporter-2.0.0.tar + docker tag deepops/prometheus-slurm-exporter:2.0.0 \ + 127.0.0.1:5000/deepops/prometheus-slurm-exporter:2.0.0 + docker push 127.0.0.1:5000/deepops/prometheus-slurm-exporter:2.0.0 + docker load -i /tmp/images/slurm-monitoring.tar + while read -r source target; do + docker tag "${source}" "${target}" + docker push "${target}" + done <<'EOF' + prom/prometheus:v3.13.0 127.0.0.1:5000/prom/prometheus:v3.13.0 + grafana/grafana:13.1.0 127.0.0.1:5000/grafana/grafana:13.1.0 + prom/alertmanager:v0.33.0 127.0.0.1:5000/prom/alertmanager:v0.33.0 + quay.io/prometheus/node-exporter:v1.11.1 127.0.0.1:5000/prometheus/node-exporter:v1.11.1 + nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless 127.0.0.1:5000/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless + EOF + ``` + +5. Extract the initialized DeepOps checkout on the provisioning machine: + + ```bash + mkdir -p "$HOME/deepops-src" + tar -xzf /path/to/import/deepops-source.tar.gz -C "$HOME/deepops-src" + cd "$HOME/deepops-src/deepops" + ``` + + Activate the provisioning environment prepared before isolation (or install + it from approved internal indexes), then copy and edit `config.example/` if + site configuration was not transferred separately. Install the mirrored + Helm version expected by `scripts/k8s/install_helm.sh` on an amd64 Linux + provisioning host so that script does not fetch its public installer: + + ```bash + mkdir -p /tmp/deepops-helm + tar -xzf /var/repos/downloads/helm-v3.17.1-linux-amd64.tar.gz \ + -C /tmp/deepops-helm + sudo install -m 0755 /tmp/deepops-helm/linux-amd64/helm /usr/local/bin/helm + helm version --short | grep '^v3.17.1' + ``` + + Mirror the matching archive for another provisioning platform; do not use + the amd64 archive there. + +6. Publish the Kubespray static-file tree, then import every Kubespray, GPU + Operator, and NFS image collected in step 3 into the registry. Do **not** run + the pinned manager's `register` action: before importing, it overwrites + `/etc/docker/daemon.json` or `/etc/containers/registries.conf`, substitutes + the importer's hostname instead of `DESTINATION_REGISTRY`, and does not back + up, restore, or restart the runtime. Extract its archive and perform the same + load/tag/push loop without changing host configuration: + + ```bash + sudo install -d /var/www/html/kubespray + sudo tar -xzf /tmp/kubespray-offline/offline-files.tar.gz \ + -C /var/www/html/kubespray + test -d /var/www/html/kubespray/offline-files + + task_kubespray_import=$(mktemp -d) + ( + set -euo pipefail + trap 'rm -rf "${task_kubespray_import}"' EXIT + tar -xzf /tmp/kubespray-offline/container-images.tar.gz \ + -C "${task_kubespray_import}" + while read -r archive repository; do + test -n "${archive}" && test -n "${repository}" + load_line=$(docker load \ + -i "${task_kubespray_import}/container-images/${archive}" | sed -n '1p') + source_ref=$(printf '%s\n' "${load_line}" | awk '{print $3}') + if [ "${source_ref}" = ID: ]; then + source_ref=$(printf '%s\n' "${load_line}" | awk '{print $4}') + fi + image_id=$(docker image inspect --format '{{.Id}}' "${source_ref}") + test -n "${image_id}" + docker tag "${image_id}" "127.0.0.1:5000/${repository}" + docker push "127.0.0.1:5000/${repository}" + done < "${task_kubespray_import}/container-images/container-images.txt" + ) + ``` + + These commands assume the fallback plain-HTTP registry is local to the + importer. For a site registry on another host, require trusted TLS or have + the site administrator configure and restart the importer's runtime under + site change control before substituting that registry endpoint. Confirm the + imported repositories through the registry's normal catalog or pull checks + before deployment. + +## 5. Configure DeepOps for internal endpoints + +Set mirror overrides in `config/group_vars/all.yml` (or a narrower group file). +Use values matching the paths actually published by the site: + +```yaml +docker_ubuntu_repo_base_url: "http://package-server/repos/download.docker.com/linux/ubuntu" +docker_ubuntu_repo_gpgkey: "http://package-server/keys/docker-ubuntu.asc" +nvidia_container_toolkit_repo_base_url: "http://package-server/repos/nvidia.github.io/libnvidia-container" +nvidia_container_toolkit_repo_gpg_url: "http://package-server/keys/libnvidia-container.asc" + +docker_rh_repo_base_url: "http://package-server/repos/docker-ce-stable" +docker_rh_repo_gpgkey: "http://package-server/keys/docker-rpm.asc" +nvidia_container_toolkit_rpm_repo_url: "http://package-server/repos/nvidia-container-toolkit.repo" + +docker_insecure_registries: + - "registry-host:5000" + +# CUDA repository mirror (required by the default Slurm path; see below) +nvidia_driver_ubuntu_cuda_keyring_url: "http://package-server/repos/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb" +nvidia_driver_rhel_cuda_repo_baseurl: "http://package-server/repos/cuda-rhel9-x86_64/" +nvidia_driver_rhel_cuda_repo_gpgkey: "http://package-server/keys/cuda-rhel9-D42D0685.pub" + +# EPEL mirror (Enterprise Linux; key import fails offline without this) +epel_package: "http://package-server/repos/epel-release-latest-9.noarch.rpm" +epel_key_url: "http://package-server/keys/RPM-GPG-KEY-EPEL-9" +``` + +For Ubuntu 24.04, prepare **every target** before running the Slurm playbook or +any play that includes `nvidia_cuda` or `nvidia_dcgm`. Replace +`package-server` with the real internal host first. These commands install the +mirrored keyring, remove the public source that package creates, write the +internal `cuda-compute-repo.list`, update that source in isolation, and then +prove the complete APT update resolves no public NVIDIA host before allowing a +normal update: + +```bash +( + set -euo pipefail + task_cuda_repo_url='http://package-server/repos/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64' + task_cuda_keyring_url="${task_cuda_repo_url}/cuda-keyring_1.1-1_all.deb" + + case "${task_cuda_repo_url}" in + *package-server*|https://developer.download.nvidia.com/*|http://developer.download.nvidia.com/*) + echo 'ERROR: set task_cuda_repo_url to the internal CUDA mirror' >&2 + exit 1 + ;; + esac + + task_cuda_keyring_deb=$(mktemp) + task_apt_uris=$(mktemp) + trap 'rm -f "${task_cuda_keyring_deb}" "${task_apt_uris}"' EXIT + curl -fsSL "${task_cuda_keyring_url}" -o "${task_cuda_keyring_deb}" + test -s "${task_cuda_keyring_deb}" + sudo dpkg -i "${task_cuda_keyring_deb}" + + if [ -e /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list ]; then + sudo mv --backup=numbered -- \ + /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list \ + /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list.disabled + fi + printf 'deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] %s /\n' \ + "${task_cuda_repo_url%/}" | + sudo tee /etc/apt/sources.list.d/cuda-compute-repo.list >/dev/null + + sudo apt-get \ + -o Dir::Etc::sourcelist='sources.list.d/cuda-compute-repo.list' \ + -o Dir::Etc::sourceparts='-' update + + apt-get --print-uris update | tee "${task_apt_uris}" + task_public_nvidia_urls=$( + sed -n "s/^'\(https\?:\/\/[^']*\)'.*/\1/p" "${task_apt_uris}" | + awk -F/ '{ host = tolower($3); sub(/:.*/, "", host) } + host == "developer.download.nvidia.com" || + host == "repo.download.nvidia.com" || + host == "nvidia.github.io" { print }' + ) + if [ -n "${task_public_nvidia_urls}" ]; then + printf 'ERROR: public NVIDIA APT URL still resolves:\n%s\n' \ + "${task_public_nvidia_urls}" >&2 + exit 1 + fi + sudo apt-get update +) +``` + +Use the matching repository directory and keyring-generated source filename on +another Ubuntu release. Keep all other APT sources internal too: the final +`apt-get update` intentionally checks the complete target configuration, which +is what the role will update. Retain the +`nvidia_driver_ubuntu_cuda_keyring_url` override so the role finds the mirrored +package; do not set `nvidia_driver_ubuntu_cuda_repo_baseurl` expecting it to +redirect APT. + +For Slurm, disable the default pull-through registry cache when upstream +Docker Hub is unreachable. If using the site's registry, the smallest honest +profile is: + +```yaml +slurm_enable_container_registry: false +standalone_container_registry_cache_enable: false +slurm_exporter_build_image: false +slurm_exporter_container: "registry-host:5000/deepops/prometheus-slurm-exporter:2.0.0" +prometheus_container: "registry-host:5000/prom/prometheus:v3.13.0" +grafana_container: "registry-host:5000/grafana/grafana:13.1.0" +alertmanager_container: "registry-host:5000/prom/alertmanager:v0.33.0" +node_exporter_container: "registry-host:5000/prometheus/node-exporter:v1.11.1" +nvidia_dcgm_container: "registry-host:5000/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless" +``` + +Mirror and override the Slurm source/archive variables listed above. If their +artifacts were deliberately excluded, disable the corresponding optional +features, for example `slurm_enable_monitoring`, `slurm_install_hpcsdk`, +`slurm_install_nhc`, `slurm_install_enroot`, or `slurm_install_pyxis`. Do not +disable a required feature merely to make a playbook pass. + +For Kubernetes, add the pinned Kubespray offline rewrites and the DeepOps chart +repositories. This layout matches the static-file extraction and image-manager +registration commands above: + +```yaml +registry_host: "registry-host:5000" +registry_addr: "registry-host:5000" +files_repo: "http://package-server/kubespray/offline-files" + +kube_image_repo: "{{ registry_host }}" +gcr_image_repo: "{{ registry_host }}" +docker_image_repo: "{{ registry_host }}" +quay_image_repo: "{{ registry_host }}" +github_image_repo: "{{ registry_host }}" +github_url: "{{ files_repo }}/github.com" +dl_k8s_io_url: "{{ files_repo }}/dl.k8s.io" +storage_googleapis_url: "{{ files_repo }}/storage.googleapis.com" +get_helm_url: "{{ files_repo }}/get.helm.sh" +local_path_provisioner_helper_image_repo: "{{ registry_host }}/busybox" + +gpu_operator_helm_repo: "http://package-server/charts" +k8s_nfs_client_helm_repo: "http://package-server/charts" + +containerd_registries_mirrors: + - prefix: "{{ registry_addr }}" + mirrors: + - host: "http://{{ registry_addr }}" + capabilities: ["pull", "resolve"] + skip_verify: true + - prefix: "nvcr.io" + mirrors: + - host: "http://{{ registry_addr }}" + capabilities: ["pull", "resolve"] + skip_verify: true + - prefix: "registry.k8s.io" + mirrors: + - host: "http://{{ registry_addr }}" + capabilities: ["pull", "resolve"] + skip_verify: true + - prefix: "docker.io" + mirrors: + - host: "http://{{ registry_addr }}" + capabilities: ["pull", "resolve"] + skip_verify: true + - prefix: "quay.io" + mirrors: + - host: "http://{{ registry_addr }}" + capabilities: ["pull", "resolve"] + skip_verify: true + - prefix: "gcr.io" + mirrors: + - host: "http://{{ registry_addr }}" + capabilities: ["pull", "resolve"] + skip_verify: true + - prefix: "ghcr.io" + mirrors: + - host: "http://{{ registry_addr }}" + capabilities: ["pull", "resolve"] + skip_verify: true +``` + +Use trusted TLS and `skip_verify: false` when the internal registry provides +it. Set `k8s_nfs_client_provisioner: false` only if the cluster uses a +site-owned storage path or intentionally has no dynamic NFS provisioner. + +The current top-level Kubernetes playbook invokes a Helm installer URL and +adds `https://charts.helm.sh/stable`; when the Ansible host platform differs +from the cluster nodes (for example a macOS or arm64 control machine), its +artifact step also downloads `kubectl` and its checksum directly from +`https://dl.k8s.io`. Run the deployment from a control host matching the +cluster platform so the playbook fetches `kubectl` from a cluster node +instead, or pre-approve an internal mirror for that URL. Its containerd +local-registry settings +also carry an explicit TODO in `config.example/group_vars/k8s_cluster.yml`. +Therefore, before declaring a fully disconnected Kubernetes run ready, prove +that these references are satisfied by approved internal endpoints or obtain a +DeepOps code change. Do not claim that the current playbook is turnkey offline. + +## 6. Preflight and deploy + +Run the machine-readable doctor from the isolated provisioning machine: + +```bash +python3 scripts/validation/deepops_doctor.py --remote --json +``` + +Require exit code `0` and top-level `"ok": true`. Review the reported host +count/groups against the approved inventory. The GPU PCI count is informational; +all other failed checks must be understood and resolved before deployment. + +### Slurm path + +```bash +ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml +``` + +Rerun the same command after a transient mirror/package error. Validate on a +Slurm controller, login, or compute node: + +```bash +python3 scripts/validation/validate_slurm.py --json +``` + +Require exit code `0`, `"ok": true`, `"nodes_unavailable": 0`, +`"gpus_configured"` greater than zero, and `"gpu_job_ok": true`. + +### Kubernetes path + +Run only after all Kubernetes package, file, chart, and image references have +been resolved to the isolated environment: + +```bash +ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml +``` + +Validate from a machine whose `kubectl` context reaches the cluster, overriding +the public default CUDA image with the imported registry image: + +```bash +python3 scripts/validation/validate_k8s.py --json --cuda-smoke \ + --cuda-image registry-host:5000/nvidia/cuda:12.4.1-base-ubuntu22.04 +``` + +Require exit code `0`, `"ok": true`, `nodes_ready == nodes_total`, +`"gpus_allocatable"` greater than zero, `"cuda_smoke_ok": true`, and +`"cuda_smoke_gpus"` greater than zero. A play recap with `failed=0` is not +the success gate. + +## Cleanup + +- On Kubernetes smoke success, the validator deletes the temporary + `deepops-validate` namespace automatically. On failure it deliberately keeps + the namespace for diagnosis; delete it only after collecting evidence: + + ```bash + kubectl delete namespace deepops-validate --ignore-not-found + ``` + +- Unmount/eject approved transfer media and remove temporary extracted copies + only after checksums, repository reachability, image pulls, and the required + validation record have been captured: + + ```bash + sudo umount /mnt/deepops-packages + sudo umount /mnt/deepops-images + sudo umount /mnt/deepops-charts + sudo umount /mnt/deepops-kubespray + sudo umount /mnt/deepops-keys + ``` + + Do not remove the internal mirrors or registry: deployed nodes continue to + need them for repairs and rescheduling. +- Remove secrets and site inventory from transfer staging. Retain the manifest, + checksums, exact DeepOps commit, mirror snapshot/version, play recap, and + validator JSON according to site policy. +- Delete `/tmp/kubespray-effective-vars.json` after artifact collection. It is a + resolved inventory export and may contain site variables that do not belong in + the transfer set or retained validation evidence. + +## Source-derived failure branches + +These branches follow from checked-in tasks, defaults, and validators; they are +not claims that this complete procedure was live-tested in an air gap. + +- **`build_offline_cache.sh` fails because + `playbooks/airgap/build-offline-cache.yml` is missing:** this automation was + removed as broken/unsupported. Use the explicit mirror workflow above. +- **Doctor reports `kubespray_submodule_initialized: false`, or Kubernetes + fails on `kubespray_defaults` imports:** transfer an archive made after + `git submodule update --init --recursive`; do not fetch from the isolated + side. +- **Generated Kubernetes lists contain a default version instead of the + configured override:** confirm that the representative host resolves the + override, regenerate its effective-variable JSON, and pass it with `-e @...`. + The generator's localhost play does not inherit `k8s_cluster` variables from + `-i` alone. +- **An image push reports an HTTPS response error against the fallback HTTP + registry:** perform the import on the registry host through + `127.0.0.1:5000`. For a remote registry, use trusted TLS or stop until the + importer's runtime is deliberately configured and restarted for that exact + endpoint; the target-node `docker_insecure_registries` value does not change + the importer. +- **Package task tries a public URL or reports a missing package:** the mirror + set or variable override is incomplete for the enabled profile. Add the exact + repository/file to the connected-side manifest, transfer a new signed + snapshot, and rerun the same idempotent playbook. +- **Slurm play reaches GitHub, SchedMD, Open MPI, or NVIDIA download hosts:** + mirror and override the applicable source URL, or intentionally disable the + optional NHC, Pyxis, HPC SDK, or monitoring component. +- **Bare `nvidia-smi` over SSH reports `No devices were found` on a Slurm + node:** DeepOps may hide GPUs from ordinary SSH sessions. Trust the + validator's `srun` job; do not diagnose the driver from the bare SSH result. +- **Slurm validator reports `gpu_job_ok: false`:** inspect its `failures` list. + If it reports driver errors, follow `skills/diagnose-driver-install/`; if the + node is down/drained, inspect its Slurm reason before resuming it. +- **Kubernetes play tries `raw.githubusercontent.com`, + `charts.helm.sh/stable`, a public chart repo, or a public image registry:** + stop. The pinned Helm/chart/image dependency is not yet served internally; + complete that mirror or gate the required DeepOps change before rerunning. +- **Kubernetes CUDA smoke is `Pending` or shows an image pull error:** confirm + the `--cuda-image` path exists in the internal registry and is reachable by + every GPU node. Inspect the retained `deepops-validate` namespace before + cleanup. +- **Kubernetes reports `gpus_allocatable: 0`:** wait for the mirrored GPU stack + to become ready, then diagnose GPU Operator/device-plugin or driver failures. + Do not use `--allow-no-gpus` as a deployment success gate for a GPU cluster. +- **A rerun uses facts from a reprovisioned/reused hostname:** add + `--flush-cache` to the same playbook command.