diff --git a/skills/README.md b/skills/README.md index 98452bf89..5d37a5355 100644 --- a/skills/README.md +++ b/skills/README.md @@ -13,6 +13,7 @@ agent (or person) can simply read the relevant `SKILL.md` before acting. |-------|----------| | [deploy-slurm-cluster](deploy-slurm-cluster/SKILL.md) | Deploying or rebuilding a Slurm GPU cluster. | | [deploy-k8s-gpu-cluster](deploy-k8s-gpu-cluster/SKILL.md) | Deploying or rebuilding a Kubernetes GPU cluster. | +| [provision-with-maas](provision-with-maas/SKILL.md) | Provisioning, tagging, validating, or releasing machines through MAAS. | | [deploy-airgapped](deploy-airgapped/SKILL.md) | Preparing mirrors and deploying Slurm or Kubernetes without Internet access. | | [validate-gpu-cluster](validate-gpu-cluster/SKILL.md) | Health checks and post-deploy verification. | | [diagnose-driver-install](diagnose-driver-install/SKILL.md) | NVIDIA driver failures, `nvidia-smi` errors, GPU pods crash-looping. | diff --git a/skills/provision-with-maas/SKILL.md b/skills/provision-with-maas/SKILL.md new file mode 100644 index 000000000..a54aa62ed --- /dev/null +++ b/skills/provision-with-maas/SKILL.md @@ -0,0 +1,252 @@ +--- +name: provision-with-maas +description: Provision or reinstall bare-metal servers and test VMs through Canonical MAAS, map deployed machines into DeepOps Ansible inventory with MAAS tags, validate access, or release them safely. Use when operating DeepOps with a MAAS-owned machine lifecycle. +--- + +# Provision with MAAS + +Keep MAAS authoritative for machine lifecycle, power, PXE/DHCP, OS images, +commissioning, deployment, release, pools, zones, and tags. Use DeepOps to +install a controller, request explicit deploy/tag/release operations, and turn +deployed machine tags into Ansible inventory. + +## Preconditions + +- Work from the DeepOps repository root on a prepared Ansible control machine. +- Confirm that every target is authorized for reinstallation. A deploy erases + its installed OS, and `maas_deploy.sh` releases every listed deployed machine + before installing it again. +- Provide a MAAS 3.x controller reachable through its REST API and an OAuth1 + API key in `consumer_key:token_key:token_secret` format. +- Ensure the provisioning VLAN can safely use MAAS DHCP, with a reserved + dynamic range that does not overlap site addresses. Targets must PXE boot on + that network. +- Configure target power control in MAAS, or arrange manual power cycles. + Commission each target to `Ready`, sync the requested Ubuntu image, and add + an SSH public key that the operator can use after deployment. +- Use Ubuntu 24.04 with the MAAS 3.7 repository for a new controller, or the + MAAS 3.5 repository while the controller remains on Ubuntu 22.04. + +## Procedure + +### Prepare MAAS and DeepOps + +1. Install DeepOps dependencies and create the private configuration if this + checkout has not been prepared: + + ```bash + ./scripts/setup.sh + ``` + +2. If DeepOps must install the MAAS controller, set `maas_adminusers`, + `maas_dns_domain`, `maas_region_controller`, + `maas_region_controller_url`, and `maas_repo` in + `config/group_vars/all.yml`. Review the inventory, then limit the + controller playbook to exactly that host: + + ```bash + ansible-playbook -l playbooks/provisioning/maas.yml + ``` + + Skip this step when using an existing MAAS controller. + +3. Edit `config/maas-inventory.yml`, which `scripts/setup.sh` creates from + `config.example/maas-inventory.yml`: + + ```yaml + api_url: "http://maas-server:5240/MAAS/api/2.0" + api_key: "consumer_key:token_key:token_secret" + ssh_user: "ubuntu" + network: "192.0.2" + ssh_bastion: "user@bastion-host" + machines: "node01 node02 node03" + ``` + + Omit `network` for first-address selection and `ssh_bastion` for direct + access. Set `machines` to the exact MAAS hostnames this operation owns; do + not rely on the script's test-VM defaults. Environment variables override + the file. `MAAS_INVENTORY_CONFIG` selects another inventory config for the + Python inventory script; the deploy script always reads the repo-local file + but accepts `MAAS_API_URL`, `MAAS_API_KEY`, `MAAS_MACHINES`, + `MAAS_SSH_USER`, `MAAS_NETWORK`, and SSH proxy overrides. + +4. Review the resolved targets and their current state before any mutation: + + ```bash + MAAS_MACHINES="node01 node02 node03" ./scripts/maas_deploy.sh --status + ``` + + Require every intended hostname to appear. Check status, IP, OS, and tags; + stop if an unexpected machine is present. + +### Map MAAS tags to inventory + +`scripts/maas_inventory.py` includes only machines whose MAAS status code is +`6` (`Deployed`). It uses each hostname as the Ansible host and maps every +MAAS tag to an Ansible group. Tag leaf groups; the script creates the DeepOps +parent/child relationships. + +Use these canonical role tags: + +| Cluster | MAAS leaf tags | +|---------|----------------| +| Kubernetes | `kube_control_plane`, `kube_node`, `etcd` | +| Slurm | `slurm-master`, `slurm-node`, optionally `slurm-nfs`, `slurm-cache`, `slurm-metric`, `slurm-login` | + +Legacy `kube-master`, `kube-node`, and `k8s-cluster` tags are accepted and +aliased to canonical group names. The deploy helper does not remove these +legacy tags when applying a profile, so remove them manually in MAAS before a +role change. Otherwise a target can remain in Kubernetes inventory groups after +switching to the Slurm profile. Other tags become same-named Ansible groups. + +For the built-in profiles, ordering is significant: the first hostname in +`machines` becomes the Kubernetes control-plane/etcd node or Slurm master; +the remaining hosts become Kubernetes or Slurm workers. Apply or change a +profile without reinstalling: + +```bash +MAAS_MACHINES="node01 node02 node03" \ + ./scripts/maas_deploy.sh --profile k8s --tags-only +``` + +Before applying the profile, this command removes the canonical Kubernetes and +Slurm test tags from every listed target. It does not remove legacy Kubernetes +tags or unrelated MAAS tags. Use MAAS directly for a custom role layout, for +example: + +```bash +maas admin tag update-nodes slurm-master add= +maas admin tag update-nodes slurm-node add= add= +``` + +### Deploy + +Run only after reviewing the exact targets and accepting a reinstall of all of +them: + +```bash +MAAS_MACHINES="node01 node02 node03" \ + ./scripts/maas_deploy.sh --os noble --profile slurm +``` + +Use `--os jammy` for Ubuntu 22.04. The default is `noble`. The script: + +1. Resolves every hostname to a MAAS system ID and stops if any is absent. +2. Releases deployed machines and waits up to 600 seconds for every machine to + reach `Ready`. +3. Requests deployment of the selected Ubuntu series and waits up to 600 + seconds for every machine to reach `Deployed`. +4. Selects an IP matching `network`, falling back to the first MAAS address, + and waits up to 120 seconds for SSH as `ssh_user`. +5. Reports `lsb_release -ds`, then clears known test tags and applies the + selected profile. + +For manual power control, perform each power-on requested by MAAS while the +script is waiting. + +### Validate after provisioning + +1. Confirm MAAS state, selected IPs, OS, and tags: + + ```bash + MAAS_MACHINES="node01 node02 node03" ./scripts/maas_deploy.sh --status + ``` + + Require every target to show `Deployed`, the requested Ubuntu release, a + usable target-network IP, and the intended role tags. + +2. Inspect the dynamic inventory before running a cluster playbook: + + ```bash + ./scripts/maas_inventory.py --list + ansible-inventory -i scripts/maas_inventory.py --graph + ``` + + This inventory is controller-wide: it includes every `Deployed` machine + visible through the configured MAAS API, not only the hostnames in + `machines` or `MAAS_MACHINES`. Require each operation-owned target to be + present with the expected leaf groups and correct first-host/worker split; + unrelated deployed machines may also appear in `all`. Inspect + `ansible_host` and `maas_*` host variables in `--list` when anything is + unexpected. + +3. Prove Ansible can reach the operation-owned machines only: + + ```bash + ansible -i scripts/maas_inventory.py 'node01:node02:node03' -m ping + ``` + + Build the host pattern from the same reviewed hostname list used for the + deploy or tag operation; do not target `all` on a shared controller. Require + `SUCCESS` and `"ping": "pong"` for every operation-owned target. This + validates provisioning and control-plane access; it does not validate a + Slurm or Kubernetes deployment. + +### Reruns and role changes + +- Rerun `--status`, inventory inspection, and Ansible ping freely; they do not + request lifecycle changes. +- Use `--tags-only` to change between the built-in Kubernetes and Slurm layouts + without reinstalling. Recheck hostname ordering first because it selects the + control-plane or master node. Inspect each target for legacy `kube-master`, + `kube-node`, or `k8s-cluster` tags and remove them manually in MAAS before the + change; `--tags-only` does not clear them. +- Treat a rerun of the deploy command as a fresh reinstall, not an idempotent + resume. It releases all listed deployed machines, including machines that + succeeded during a partially failed multi-machine run. +- A released machine that is still commissioned and `Ready` can be deployed + again without commissioning it again. + +### Cleanup + +Release only the exact machines owned by the operation: + +```bash +MAAS_MACHINES="node01 node02 node03" ./scripts/maas_deploy.sh --release +``` + +Require `All machines released.` and verify `--status` shows `Ready`. Released +machines disappear from dynamic inventory because it includes only `Deployed` +machines. Release does not clear their MAAS tags; remove residual role tags in +MAAS separately if the machines are being reassigned, after confirming tag +ownership. + +## Failure branches + +- **Inventory returns only an empty `_meta` object without an error:** the + configuration is missing or still contains example placeholders. Populate + `api_url` and `api_key`, or set the matching environment variables, then + rerun `--list`. +- **`ERROR: MAAS_API_URL not configured` or `ERROR: MAAS_API_KEY not + configured`:** populate the missing deploy configuration or its environment + override. The deploy helper does not use the inventory helper's graceful + empty-inventory behavior. +- **`MAAS API error: 401`:** replace or correct `MAAS_API_KEY`. If the deploy + helper reports that the key must use + `consumer_key:token_key:token_secret`, correct its three-part format before + making another request. +- **`MAAS connection error`:** verify `MAAS_API_URL` and reachability from the + control machine. Do not continue to lifecycle operations until inventory or + status reads succeed. +- **`Machine '' not found in MAAS`:** compare `machines` or + `MAAS_MACHINES` with MAAS hostnames. Correct the target list; do not replace + it with a broader list. +- **Inventory is valid but a target is absent:** only `Deployed` machines with + a non-empty hostname are included. Check `--status`; finish deployment, or + expect a released/`Ready` machine to remain absent. +- **Inventory chooses the wrong address or SSH times out after 120 seconds:** + set `network`/`MAAS_NETWORK` to the target subnet, verify `ssh_user`, the + imported MAAS SSH key, and `ssh_bastion` or proxy settings, then rerun the + read-only ping checks. A machine with no MAAS address cannot yield a usable + `ansible_host`. +- **A machine does not reach `Ready` or `Deployed` within 600 seconds:** inspect + its MAAS state and events. For manual power, perform the requested power + cycle. If PXE never begins, verify PXE boot and that DHCP is enabled on the + correct VLAN with a non-overlapping dynamic range. Rerun deployment only + after deciding whether reinstalling every listed target is acceptable. +- **`--tags-only` fails immediately:** supply `--profile k8s` or + `--profile slurm`; other profile names are rejected. If groups are still + wrong, inspect tag names and target ordering. Remove legacy `kube-master`, + `kube-node`, and `k8s-cluster` tags manually in MAAS because reapplying a + profile will not clear them; then reapply the intended profile and regenerate + inventory.