diff --git a/src/content/docs/platform/self-hosting/managed-docker.mdx b/src/content/docs/platform/self-hosting/managed-docker.mdx index 2ee278a3..2bc92cd3 100644 --- a/src/content/docs/platform/self-hosting/managed-docker.mdx +++ b/src/content/docs/platform/self-hosting/managed-docker.mdx @@ -188,25 +188,9 @@ oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker" ## Private Docker registries -The worker automatically uses credentials from your Docker config (`~/.docker/config.json`) when pulling task images. If your [environments](/platform/environments/) use images from a private registry, authenticate the worker's host first: +The Docker backend reads registry credentials from the worker's Docker config. You can also override the Warp Agent sidecar image when the worker host must pull it from an internal registry. -```bash -docker login your-registry.example.com -``` - -When running the worker via Docker, mount the Docker config into the container: - -```bash -docker run \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v ~/.docker/config.json:/root/.docker/config.json:ro \ - -e WARP_API_KEY="$WARP_API_KEY" \ - warpdotdev/oz-agent-worker --worker-id "my-worker" -``` - -:::note -Sidecar images (the `oz` binary and dependencies) are pulled from public registries and do not require authentication. -::: +See [Pulling self-hosted images from a private registry](/platform/self-hosting/private-container-registry/) for image mirroring, authentication, and worker configuration. --- @@ -221,5 +205,6 @@ Once your Docker worker is connected, route tasks to it with `--host "- + Mirror self-hosted worker images into a private registry and configure Docker + or Kubernetes workers to pull from it. +--- + +Mirror the images used by self-hosted workers when compute hosts cannot pull directly from a public registry. Your registry becomes the pull source for the worker process, task environment, Warp Agent sidecar, and Kubernetes preflight job. + +## Images to mirror + +Inventory the images used by each worker pool before blocking public-registry egress. + +| Image | Purpose | Configuration | +| --- | --- | --- | +| `warpdotdev/oz-agent-worker` | Long-lived worker daemon | Docker run image or Helm `image.repository` | +| Environment or default image | Main task filesystem and toolchain | Warp environment image or Kubernetes `defaultImage` | +| `warpdotdev/warp-agent` | Warp Agent runtime mounted at `/agent` | `sidecar_image` or Helm `kubernetesBackend.sidecarImage` | +| `busybox:1.36` | Kubernetes startup preflight | `preflight_image` or Helm `kubernetesBackend.preflightImage` | + +Pin the worker image to the immutable timestamp tag or digest from the [worker release](https://github.com/warpdotdev/oz-agent-worker/releases). Keep the mirrored Warp Agent sidecar current with the source image. Warp normally sends a version-matched sidecar reference with each task; a static `sidecar_image` override replaces that reference, so an outdated mirror can become incompatible with a newer worker or task. + +The worker does not provide a stable pre-dispatch method to discover the exact version-matched Warp Agent tag. Ask your Warp account team which tag to mirror, and validate that tag when you update the worker. + +:::caution +The Docker backend can override only the Warp Agent sidecar. It cannot rewrite other server-provided sidecars used by optional capabilities. The Kubernetes config file can override Claude Code and Codex sidecars with `coding_cli_sidecars`, but the current Helm chart does not expose that field. A deployment that blocks all public-registry access may not support optional harness or Computer Use runs until every required sidecar has a supported override. +::: + +## Copying images into the registry + +Use a registry copy tool that preserves all image architectures. The following example uses [Skopeo](https://github.com/containers/skopeo). + +```bash +export PRIVATE_REGISTRY="registry.internal.example.com/warp" +export WORKER_TAG="YOUR_WORKER_RELEASE_TAG" +export WARP_AGENT_TAG="YOUR_APPROVED_WARP_AGENT_TAG" +skopeo login registry.internal.example.com + +skopeo copy --all \ + "docker://docker.io/warpdotdev/oz-agent-worker:${WORKER_TAG}" \ + "docker://${PRIVATE_REGISTRY}/oz-agent-worker:${WORKER_TAG}" + +skopeo copy --all \ + "docker://docker.io/warpdotdev/warp-agent:${WARP_AGENT_TAG}" \ + "docker://${PRIVATE_REGISTRY}/warp-agent:${WARP_AGENT_TAG}" + +skopeo copy --all \ + "docker://docker.io/library/busybox:1.36" \ + "docker://${PRIVATE_REGISTRY}/busybox:1.36" + +skopeo copy --all \ + "docker://docker.io/library/ubuntu:22.04" \ + "docker://${PRIVATE_REGISTRY}/agent-base:22.04" +``` + +Replace `YOUR_APPROVED_WARP_AGENT_TAG` with the source tag your organization has validated for the deployment. The example mirrors Ubuntu as the task image. Replace that source with your own task image, then set the private image reference on the [Warp environment](/platform/environments/) used by the worker pool. + +## Configuring the Docker backend + +Authenticate the host Docker client so it can pull the mirrored worker image: + +```bash +docker login registry.internal.example.com +``` + +Configure the mirrored Warp Agent sidecar and a pull policy: + +```yaml title="worker.yaml" +worker_id: "private-registry-docker" +backend: + docker: + image_pull_policy: "IfNotPresent" + sidecar_image: "registry.internal.example.com/warp/warp-agent:WARP_AGENT_TAG" +``` + +The published worker image runs as the non-root `oz` user with UID `10001`. If the worker runs as a container, create a dedicated Docker config that this UID can read: + +```bash +export WORKER_DOCKER_CONFIG="/var/lib/oz-agent-worker/docker-config" + +sudo install -d -m 0700 -o 10001 "$WORKER_DOCKER_CONFIG" +sudo docker --config "$WORKER_DOCKER_CONFIG" \ + login registry.internal.example.com +sudo chown 10001 "$WORKER_DOCKER_CONFIG/config.json" +sudo chmod 0400 "$WORKER_DOCKER_CONFIG/config.json" +``` + +The worker also needs read and write access to the Docker daemon socket. This example supports a rootful Docker daemon on Linux whose socket grants read and write access to its group. Capture that numeric group ID: + +```bash +export DOCKER_SOCKET_GID="$(stat -c '%g' /var/run/docker.sock)" +stat -c '%A %g %n' /var/run/docker.sock +``` + +The group permission bits in the `stat` output must include `rw`. Start the worker with that group as a supplemental group: + +```bash +docker run \ + --group-add "$DOCKER_SOCKET_GID" \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + --volume "$WORKER_DOCKER_CONFIG:/home/oz/.docker:ro" \ + --volume "$PWD/worker.yaml:/etc/oz-agent-worker/config.yaml:ro" \ + --env DOCKER_CONFIG=/home/oz/.docker \ + --env WARP_API_KEY="$WARP_API_KEY" \ + "registry.internal.example.com/warp/oz-agent-worker:WORKER_TAG" \ + --config-file /etc/oz-agent-worker/config.yaml +``` + +Replace `WORKER_TAG` and `WARP_AGENT_TAG` with the mirrored tags. The task image must also point to the private registry through its Warp environment; the worker does not rewrite task image registry names. Use a host process or a Docker endpoint with its own access controls when the daemon does not expose a group-readable and group-writable Linux socket. + +## Configuring the Kubernetes backend + +Create one pull secret for the worker Deployment and task Jobs: + +```bash +kubectl create namespace warp-oz \ + --dry-run=client \ + --output yaml | kubectl apply --filename - +read -r -p "Registry username: " REGISTRY_USERNAME +read -r -s -p "Registry token: " REGISTRY_TOKEN +printf '\n' +kubectl create secret docker-registry warp-registry \ + --namespace warp-oz \ + --docker-server registry.internal.example.com \ + --docker-username "$REGISTRY_USERNAME" \ + --docker-password "$REGISTRY_TOKEN" +unset REGISTRY_USERNAME REGISTRY_TOKEN +``` + +Set the worker, task, sidecar, and preflight image references in a Helm values file: + +```yaml title="private-registry-values.yaml" +image: + repository: registry.internal.example.com/warp/oz-agent-worker + tag: WORKER_TAG + pullPolicy: IfNotPresent + pullSecrets: + - name: warp-registry + +kubernetesBackend: + defaultImage: registry.internal.example.com/warp/agent-base:22.04 + preflightImage: registry.internal.example.com/warp/busybox:1.36 + sidecarImage: registry.internal.example.com/warp/warp-agent:WARP_AGENT_TAG + podTemplate: + imagePullSecrets: + - name: warp-registry +``` + +Install the chart with those values: + +```bash +git clone --branch YOUR_WORKER_RELEASE_TAG --depth 1 \ + https://github.com/warpdotdev/oz-agent-worker.git + +helm upgrade --install oz-agent-worker ./oz-agent-worker/charts/oz-agent-worker \ + --namespace warp-oz \ + --create-namespace \ + --set worker.workerId=private-registry-kubernetes \ + --values private-registry-values.yaml +``` + +`image.pullSecrets` authenticates the long-lived worker Deployment. `kubernetesBackend.podTemplate.imagePullSecrets` authenticates task Jobs and the startup preflight Job. + +A Warp environment image takes precedence over `kubernetesBackend.defaultImage`. If the run uses an environment, update that environment to the mirrored task image instead of relying on `defaultImage`. + +## Verifying registry isolation + +Start with one test run that uses the core Warp Agent and no optional sidecars: + +```bash +oz agent run-cloud \ + --host "private-registry-kubernetes" \ + --prompt "Print the operating system release and exit." +``` + +Confirm the worker logs show the private task and sidecar references. In Kubernetes, inspect the task Pod: + +```bash +kubectl get pods --namespace warp-oz +kubectl get pod TASK_POD --namespace warp-oz \ + --output jsonpath='{range .spec.initContainers[*]}{.image}{"\n"}{end}{range .spec.containers[*]}{.image}{"\n"}{end}' +``` + +Block public-registry egress only after the worker, preflight Job, and task Pod all use private references. Then repeat the test for every harness and optional capability allowed in the worker pool. + +## Troubleshooting + +**Docker reports `pull access denied`**\ +Run `docker login` as the worker's OS account. If the worker runs in Docker, confirm `DOCKER_CONFIG` points to `/home/oz/.docker` and the mounted `config.json` is readable by UID `10001`. + +**Docker reports `permission denied` for `/var/run/docker.sock`**\ +Confirm the socket's group has read and write access, then pass its numeric group ID to the worker with `--group-add`. + +**Kubernetes reports `ImagePullBackOff`**\ +Confirm `warp-registry` exists in the task namespace. The worker Deployment needs `image.pullSecrets`, while task and preflight Pods need `podTemplate.imagePullSecrets`. + +**The task still pulls from Docker Hub**\ +Check the Warp environment image. Environment images take precedence over the Kubernetes default, and the worker does not rewrite their registry host. + +## Related pages + +* [Managed: Docker backend](/platform/self-hosting/managed-docker/) — Configure Docker daemon access and task containers. +* [Managed: Kubernetes backend](/platform/self-hosting/managed-kubernetes/) — Configure the Helm chart, task Pod template, and preflight job. +* [Environments](/platform/environments/) — Set the task image used by self-hosted runs. +* [Security and networking](/platform/self-hosting/security-and-networking/) — Review image egress and other network requirements. diff --git a/src/content/docs/platform/self-hosting/reference.mdx b/src/content/docs/platform/self-hosting/reference.mdx index c5ced7a2..b26e873f 100644 --- a/src/content/docs/platform/self-hosting/reference.mdx +++ b/src/content/docs/platform/self-hosting/reference.mdx @@ -143,6 +143,8 @@ backend: **`backend.docker`:** * `volumes` — List of volume mounts (same format as `-v` flag). +* `image_pull_policy` — One of `Always`, `Never`, or `IfNotPresent`. Defaults to `Always`. +* `sidecar_image` — Image reference that replaces the server-provided Warp Agent sidecar mounted at `/agent`. It does not replace additional sidecars. * `environment` — List of environment variables with `name` and optional `value`. If `value` is omitted, the variable is inherited from the host. **`backend.kubernetes`:** @@ -152,6 +154,8 @@ backend: * `default_image` — Default Docker image for task Jobs when the run has no Warp environment image. Precedence: Warp environment image > `default_image` > `ubuntu:22.04`. Set this to skip creating a Warp environment when all your tasks use the same base image. * `image_pull_policy` — One of `Always`, `Never`, or `IfNotPresent`. Defaults to `IfNotPresent`. * `preflight_image` — Image used for the startup preflight Job. Defaults to `busybox:1.36`. Override this if your cluster only allows pulling from an internal or allowlisted registry. +* `sidecar_image` — Image reference that replaces the server-provided Warp Agent sidecar mounted at `/agent`. +* `coding_cli_sidecars` — Map of harness names such as `claude` or `codex` to custom coding CLI sidecar images. This config-file field is not exposed by the current Helm chart. * `setup_command` — Shell command to run before each task. * `teardown_command` — Shell command to run after each task completes. * `extra_labels` — Map of additional labels to add to task Jobs and Pods. @@ -199,6 +203,7 @@ Once a worker is running, route cloud agent runs to it with the `--host` flag or * [Managed: Docker](/platform/self-hosting/managed-docker/) — Docker backend setup, connectivity, and private registries. * [Managed: Kubernetes](/platform/self-hosting/managed-kubernetes/) — Kubernetes backend setup, Helm chart, pod template, and operational notes. +* [Private container registry](/platform/self-hosting/private-container-registry/) — Mirror and pull worker, task, sidecar, and preflight images from an internal registry. * [Managed: Direct](/platform/self-hosting/managed-direct/) — Direct backend setup and workspace model. * [Self-hosting overview](/platform/self-hosting/) — Architecture, decision guide, and Enterprise requirements. * [Environments](/platform/environments/) — Define the Docker image, repos, and setup commands used by task containers. diff --git a/src/sidebar.ts b/src/sidebar.ts index 5115ad6e..c92bc92a 100644 --- a/src/sidebar.ts +++ b/src/sidebar.ts @@ -657,6 +657,7 @@ export const sidebarTopics: StarlightSidebarTopicsUserConfig = [ { slug: 'platform/self-hosting/quickstart', label: 'Self-hosting quickstart' }, { slug: 'platform/self-hosting/managed-docker', label: 'Managed: Docker' }, { slug: 'platform/self-hosting/managed-kubernetes', label: 'Managed: Kubernetes' }, + { slug: 'platform/self-hosting/private-container-registry', label: 'Private container registry' }, { slug: 'platform/self-hosting/managed-direct', label: 'Managed: Direct' }, { slug: 'platform/self-hosting/unmanaged', label: 'Unmanaged' }, 'platform/self-hosting/monitoring',