diff --git a/docs/architecture.md b/docs/architecture.md index 75b90dc..b83e499 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -125,6 +125,9 @@ and setup-script hash. `POST /templates` creates a `building` record and returns `202 Accepted`. Callers poll until the template becomes `available` or `failed`. Templates outlive hosts. Each provider builds and deletes its own templates behind `TemplateCapability`. +OCI builds use one shared image repository when configured. Labels describe +purpose; unique build tags identify publications. Published template images +use repository digests. See [registry configuration](deploy.md#shared-template-image-repository). A host request can name an available template by its ID — the ID that the create returned. The template's image becomes the host image. An diff --git a/docs/deploy.md b/docs/deploy.md index 5cbc55e..725000e 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -306,6 +306,10 @@ Core, optional: | `SERVICE_LABEL` | `drukbox` | Label stamped onto provider resources (VM tags, SG tags). | | `UVICORN_HOST` | `0.0.0.0` | API bind address. Set `127.0.0.1` to restrict to loopback. | | `PROVISIONING_GRACE_SECONDS` | `600` | Safety TTL on in-flight hosts so the janitor reaps row + VM if the client disconnects mid-provision. Must exceed the worst-case provision duration. | +| `REGISTRY_HOST` | — | Registry host for private images, such as `ghcr.io` or `docker.io`. | +| `TEMPLATE_REPOSITORY` | — | Shared template repository path within `REGISTRY_HOST`, with no tag or digest. | +| `REGISTRY_USERNAME` | — | Registry user for template pushes and private exe pulls. | +| `REGISTRY_PASSWORD` | — | Registry password or token. | | `TEMPLATE_BUILD_TIMEOUT` | `3600` | Max age in seconds of an unfinished template build before the janitor marks it failed. | | `TEMPLATE_FAILED_RETENTION` | `86400` | Seconds that failed template records and diagnostics remain before the janitor deletes them. | | `TEMPLATE_UNUSED_TTL` | `1209600` | Seconds that an available template remains after its last use, or creation when never used. | @@ -334,9 +338,6 @@ exe.dev provider: | --- | --- | --- | | `EXE_API_TOKEN` | — (required) | Bearer token for the exe.dev exec API. | | `EXE_DEFAULT_IMAGE` | — (required) | Image used when the caller omits `image`. | -| `EXE_IMAGE_REGISTRY` | — | Repository prefix for derived template images. A VM created from this registry gets `--registry-auth` so exe.dev can pull a private image. | -| `EXE_REGISTRY_USERNAME` | — | Username for the derived-template image registry. | -| `EXE_REGISTRY_PASSWORD` | — | Password or token for the derived-template image registry. | | `EXE_API_URL` | `https://exe.dev` | API base URL. | | `EXE_API_TIMEOUT` | `30.0` | Timeout for exe.dev API calls. | | `EXE_BOOTSTRAP_SSH_TIMEOUT_SECONDS` | `30.0` | ssh-keyscan retry budget for a fresh exe.dev sandbox. | @@ -422,3 +423,51 @@ The published image does not contain the `sbx` CLI. Mount the binary and the auth store of the host, as [Local microVMs with Docker Sandboxes](#local-microvms-with-docker-sandboxes) shows. Set `DOCKER_SANDBOXES_API` to the mounted daemon socket. + +## Shared template image repository + +Registry access applies to private images, including boot images that are +not templates. Set the registry host and credentials together. Set a template +repository separately when this installation must publish template images: + +```dotenv +REGISTRY_HOST=ghcr.io +TEMPLATE_REPOSITORY=acme/sandbox-templates +REGISTRY_USERNAME=builder +REGISTRY_PASSWORD= +``` + +For Docker Hub, set `REGISTRY_HOST=docker.io`. Registry access does not +require `TEMPLATE_REPOSITORY`. Publishing templates requires a repository +and a credential that can push to it. Keep the credentials in +the service environment. Do not send them in a template request. + +The caller supplies a descriptive `label`, such as `site-builder-build`. +Drukbox converts it to lowercase letters, digits, and hyphens. Each build +gets a unique UUID suffix. A tag has this form: + +```text +ghcr.io/acme/sandbox-templates:site-builder-build-01992000123470008000123456789abc +``` + +After the push, Drukbox stores the tag and digest together in the template's +`image` field: `ghcr.io/acme/sandbox-templates:-@sha256:`. +Hosts use that digest. The tag identifies the build for local image cleanup. +The label does not change template identity: the provider, base image, and setup script hash +still determine reuse. A repeated request returns the existing record. +To retry a failed build, delete its template record, then create it again. + +The `exe` provider sends pull credentials for boot images on `REGISTRY_HOST`, +including images outside the template repository. Credentials never go to a +different registry host. Template builds also require `TEMPLATE_REPOSITORY`. `docker` and +`docker-sbx` publish to it when configured. Without a template destination, +they keep their template images local. Native VM image providers do not use +this OCI destination. The `docker-sbx` daemon has its own image store and +registry access. Configure its private pulls or load the template as described +in [local microVM setup](#local-microvms-with-docker-sandboxes). +Shared push credentials do not configure that daemon. + +All templates share repository access and retention policy. The janitor +removes template records and local images. It does not delete remote registry +manifests. Configure registry retention separately, and retain images that +active templates still reference. diff --git a/docs/security.md b/docs/security.md index ca9c314..b4da892 100644 --- a/docs/security.md +++ b/docs/security.md @@ -76,7 +76,7 @@ covered in [Networking](networking.md). The security-relevant summary: ## Secrets and in-VM metadata -Provider tokens (`EXE_API_TOKEN`, `EXE_REGISTRY_PASSWORD`, +Provider tokens (`EXE_API_TOKEN`, `REGISTRY_PASSWORD`, `HETZNER_API_TOKEN`, Tailscale OAuth) and AWS credentials are read from the environment / the AWS SDK default chain and never written to the database or returned by the API. Host secret recipes are encrypted in diff --git a/src/core/settings.py b/src/core/settings.py index 0bccdd2..1bc84fe 100644 --- a/src/core/settings.py +++ b/src/core/settings.py @@ -1,7 +1,7 @@ from functools import lru_cache -from typing import Annotated +from typing import Annotated, Self -from pydantic import BeforeValidator, Field, SecretStr +from pydantic import BeforeValidator, Field, SecretStr, model_validator from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict from sqlalchemy_encrypted_field import validate_keys @@ -77,6 +77,30 @@ class Settings(BaseSettings): validation_alias="PROVISIONING_GRACE_SECONDS", description="Safety TTL on the host row while provisioning is in flight.", ) + registry_host: str = Field( + default="", + validation_alias="REGISTRY_HOST", + pattern=r"^$|^[a-z0-9.-]+(?::[0-9]+)?$", + description="Registry host for private images, such as ghcr.io or docker.io.", + ) + template_repository: str = Field( + default="", + validation_alias="TEMPLATE_REPOSITORY", + pattern=( + r"^$|^[a-z0-9]+(?:(?:[._]|__|-+)[a-z0-9]+)*" + r"(?:/[a-z0-9]+(?:(?:[._]|__|-+)[a-z0-9]+)*)*$" + ), + description="Template repository path within REGISTRY_HOST, without a tag.", + ) + registry_username: str = Field( + default="", + validation_alias="REGISTRY_USERNAME", + ) + registry_password: SecretStr = Field( + default=SecretStr(""), + validation_alias="REGISTRY_PASSWORD", + ) + template_build_timeout: int = Field( default=3600, gt=0, @@ -135,6 +159,18 @@ class Settings(BaseSettings): description="Upper bound on pool-maintainer provisions per tick, across all providers.", ) + @model_validator(mode="after") + def validate_registry(self) -> Self: + values = { + "REGISTRY_HOST": self.registry_host, + "REGISTRY_USERNAME": self.registry_username, + "REGISTRY_PASSWORD": self.registry_password.get_secret_value(), + } + if (self.template_repository or any(values.values())) and not all(values.values()): + missing = ", ".join(name for name, value in values.items() if not value) + raise ValueError(f"Registry is incomplete. Set: {missing}") + return self + def get_pool_targets(self) -> dict[str, int]: # POOL_SIZE seeds the default provider's target and POOL_SIZES # overrides per provider; providers at zero drop out entirely. diff --git a/src/core/tests/test_settings.py b/src/core/tests/test_settings.py index 5e7e245..178672f 100644 --- a/src/core/tests/test_settings.py +++ b/src/core/tests/test_settings.py @@ -185,3 +185,102 @@ def test_load_test_env_overrides_ambient_values(monkeypatch: pytest.MonkeyPatch) monkeypatch.setenv("TAILSCALE_ENABLED", "false") conftest.load_test_env() assert os.environ["TAILSCALE_ENABLED"] == "true" + + +@pytest.mark.parametrize("host", ["ghcr.io", "docker.io", "registry.example:5000"]) +def test_registry_access_does_not_require_templates(monkeypatch, host): + settings = _settings_with( + monkeypatch, + { + **_base_env(), + "REGISTRY_HOST": host, + "REGISTRY_USERNAME": "builder", + "REGISTRY_PASSWORD": "private-token", + "TEMPLATE_REPOSITORY": "", + }, + ) + assert settings.registry_host == host + assert settings.template_repository == "" + assert settings.registry_password.get_secret_value() == "private-token" + assert "private-token" not in repr(settings) + + +@pytest.mark.parametrize("repository", ["acme/templates", "org/team/templates"]) +def test_template_destination_uses_registry_access(monkeypatch, repository): + settings = _settings_with( + monkeypatch, + { + **_base_env(), + "REGISTRY_HOST": "ghcr.io", + "REGISTRY_USERNAME": "builder", + "REGISTRY_PASSWORD": "private-token", + "TEMPLATE_REPOSITORY": repository, + }, + ) + assert settings.template_repository == repository + + +@pytest.mark.parametrize("host", ["https://ghcr.io", "ghcr.io/acme", "ghcr.io@evil.example"]) +def test_registry_host_rejects_url_and_path(monkeypatch, host): + with pytest.raises(ValueError, match="REGISTRY_HOST"): + _settings_with( + monkeypatch, + { + **_base_env(), + "REGISTRY_HOST": host, + "REGISTRY_USERNAME": "builder", + "REGISTRY_PASSWORD": "private-token", + }, + ) + + +@pytest.mark.parametrize( + "repository", + [ + "https://ghcr.io/acme/templates", + "acme/templates:latest", + "acme/templates@sha256:abc", + "acme/", + ], +) +def test_template_repository_rejects_url_tag_or_digest(monkeypatch, repository): + with pytest.raises(ValueError, match="TEMPLATE_REPOSITORY"): + _settings_with( + monkeypatch, + { + **_base_env(), + "REGISTRY_HOST": "ghcr.io", + "REGISTRY_USERNAME": "builder", + "REGISTRY_PASSWORD": "private-token", + "TEMPLATE_REPOSITORY": repository, + }, + ) + + +def test_partial_registry_names_missing_setting_without_secret(monkeypatch): + with pytest.raises(ValueError) as error: + _settings_with( + monkeypatch, + { + **_base_env(), + "REGISTRY_HOST": "ghcr.io", + "REGISTRY_USERNAME": "", + "REGISTRY_PASSWORD": "private-token", + }, + ) + assert "REGISTRY_USERNAME" in str(error.value) + assert "private-token" not in str(error.value) + + +def test_template_destination_requires_registry_access(monkeypatch): + with pytest.raises(ValueError, match="REGISTRY_HOST"): + _settings_with( + monkeypatch, + { + **_base_env(), + "REGISTRY_HOST": "", + "REGISTRY_USERNAME": "", + "REGISTRY_PASSWORD": "", + "TEMPLATE_REPOSITORY": "acme/templates", + }, + ) diff --git a/src/providers/docker/api.py b/src/providers/docker/api.py index 774ac04..5d229af 100644 --- a/src/providers/docker/api.py +++ b/src/providers/docker/api.py @@ -116,15 +116,30 @@ async def push_image( *, username: str | None = None, password: str | None = None, - ) -> None: + ) -> str: # Credentials travel per call as an X-Registry-Auth header; the # global docker credential store is never touched. auth = {"username": username, "password": password} if username and password else None try: await self._get_client().images.push(image, auth=auth) + metadata = await self._get_client().images.inspect(image) except (aiodocker.DockerError, aiohttp.ClientError) as exc: raise DockerTransportError(_detail(exc)) from exc + repository = image.rpartition(":")[0] + # Docker omits docker.io and its library namespace in RepoDigests. + digest_repository = repository + if repository.startswith("docker.io/"): + digest_repository = repository.removeprefix("docker.io/").removeprefix("library/") + digests = [ + digest.partition("@")[2] + for digest in metadata.get("RepoDigests") or [] + if digest.startswith((f"{repository}@sha256:", f"{digest_repository}@sha256:")) + ] + if len(digests) != 1: + raise DockerTransportError(f"Pushed image {image!r} has no unique repository digest") + return f"{image}@{digests[0]}" + async def server_version(self) -> str: try: version = await self._get_client().version() diff --git a/src/providers/docker/images.py b/src/providers/docker/images.py index 4d69737..19e47c9 100644 --- a/src/providers/docker/images.py +++ b/src/providers/docker/images.py @@ -1,7 +1,10 @@ -import hashlib import io +import re import tarfile +from uuid6 import uuid7 + +from core.settings import get_settings from providers.exceptions import ProviderNotFoundError, ProviderTransportError from .api import DockerAPI @@ -10,13 +13,11 @@ def derive_image_name( *, - base_image: str, - setup_script: str, + label: str, repository: str = "drukbox-template", ) -> str: - identity = base_image.encode("utf-8") + b"\0" + setup_script.encode("utf-8") - digest = hashlib.sha256(identity).hexdigest()[:12] - return f"{repository}:{digest}" + purpose = re.sub(r"[^a-z0-9]+", "-", label.lower()).strip("-")[:95].rstrip("-") + return f"{repository}:{purpose or 'template'}-{uuid7().hex}" def create_build_context(*, base_image: str, setup_script: str) -> bytes: @@ -41,16 +42,26 @@ async def build_derived_image( *, base_image: str, setup_script: str, - repository: str = "drukbox-template", + label: str, ) -> str: + settings = get_settings() image = derive_image_name( - base_image=base_image, - setup_script=setup_script, - repository=repository, + label=label, + repository=( + f"{settings.registry_host}/{settings.template_repository}" + if settings.template_repository + else "drukbox-template" + ), ) context_tar = create_build_context(base_image=base_image, setup_script=setup_script) try: await docker.build_image(image, context_tar) + if settings.template_repository: + return await docker.push_image( + image, + username=settings.registry_username, + password=settings.registry_password.get_secret_value(), + ) except DockerProviderError as exc: raise ProviderTransportError(str(exc)) from exc return image @@ -58,7 +69,9 @@ async def build_derived_image( async def remove_derived_image(docker: DockerAPI, image: str) -> None: try: - await docker.remove_image(image) + # Keep the unique build tag in the pinned reference so cleanup removes + # this build's tag, even when another build has the same digest. + await docker.remove_image(image.partition("@")[0]) except DockerImageNotFoundError as exc: raise ProviderNotFoundError(f"docker image '{image}' was not found") from exc except DockerProviderError as exc: diff --git a/src/providers/docker/provider.py b/src/providers/docker/provider.py index c5e7087..e10906e 100644 --- a/src/providers/docker/provider.py +++ b/src/providers/docker/provider.py @@ -140,6 +140,7 @@ async def build_template_image( self.api, base_image=base_image, setup_script=setup_script, + label=label, ) async def delete_template_image(self, image: str) -> None: diff --git a/src/providers/docker/tests/test_api.py b/src/providers/docker/tests/test_api.py index d07aa07..fc3fa4c 100644 --- a/src/providers/docker/tests/test_api.py +++ b/src/providers/docker/tests/test_api.py @@ -30,6 +30,9 @@ def _fake_docker(**overrides: object) -> SimpleNamespace: build=AsyncMock(), delete=AsyncMock(), push=AsyncMock(), + inspect=AsyncMock( + return_value={"RepoDigests": ["ghcr.io/acme/template@sha256:" + "a" * 64]} + ), ), version=AsyncMock(return_value={"Version": "29.6.2"}), close=AsyncMock(), @@ -137,10 +140,12 @@ async def test_missing_image_maps_to_not_found() -> None: async def test_push_image_sends_per_call_credentials() -> None: fake = _fake_docker() - await _api(fake).push_image( + image = await _api(fake).push_image( "ghcr.io/acme/template:tag", username="builder", password="registry-secret" ) + assert image == "ghcr.io/acme/template:tag@sha256:" + "a" * 64 + fake.images.inspect.assert_awaited_once_with("ghcr.io/acme/template:tag") fake.images.push.assert_awaited_once_with( "ghcr.io/acme/template:tag", auth={"username": "builder", "password": "registry-secret"}, @@ -178,3 +183,41 @@ async def test_aclose_closes_the_client_once_created() -> None: await api.aclose() fake.close.assert_awaited_once_with() + + +@pytest.mark.parametrize("digests", [None, [], ["ghcr.io/other/template@sha256:" + "a" * 64]]) +async def test_push_rejects_missing_repository_digest(digests): + fake = _fake_docker() + fake.images.inspect.return_value = {"RepoDigests": digests} + + with pytest.raises(DockerTransportError, match="no unique repository digest"): + await _api(fake).push_image("ghcr.io/acme/template:tag") + + +async def test_push_selects_digest_for_the_pushed_repository(): + fake = _fake_docker() + expected = "ghcr.io/acme/template@sha256:" + "a" * 64 + fake.images.inspect.return_value = { + "RepoDigests": ["ghcr.io/other/template@sha256:" + "b" * 64, expected] + } + + assert await _api(fake).push_image("ghcr.io/acme/template:tag") == expected.replace( + "@", ":tag@" + ) + + +@pytest.mark.parametrize( + ("repository", "stored_repository"), + [ + ("docker.io/acme/templates", "acme/templates"), + ("docker.io/library/template", "template"), + ("registry.example:5000/acme/templates", "registry.example:5000/acme/templates"), + ], +) +async def test_push_returns_full_repository_with_engine_digest(repository, stored_repository): + fake = _fake_docker() + fake.images.inspect.return_value = {"RepoDigests": [f"{stored_repository}@sha256:" + "a" * 64]} + + assert ( + await _api(fake).push_image(f"{repository}:tag") == f"{repository}:tag@sha256:" + "a" * 64 + ) diff --git a/src/providers/docker/tests/test_images.py b/src/providers/docker/tests/test_images.py index a2766a8..90e298a 100644 --- a/src/providers/docker/tests/test_images.py +++ b/src/providers/docker/tests/test_images.py @@ -1,7 +1,11 @@ import io +import re import tarfile +from unittest.mock import AsyncMock, MagicMock -from providers.docker.images import create_build_context, derive_image_name +import pytest + +from providers.docker.images import create_build_context, derive_image_name, remove_derived_image def test_create_build_context_contains_the_base_and_verbatim_script() -> None: @@ -22,12 +26,31 @@ def test_create_build_context_contains_the_base_and_verbatim_script() -> None: ) -def test_derive_image_name_is_deterministic_and_base_specific() -> None: - first = derive_image_name(base_image="sandbox:base", setup_script="apt-get update") - repeated = derive_image_name(base_image="sandbox:base", setup_script="apt-get update") - different_base = derive_image_name(base_image="sandbox:other", setup_script="apt-get update") +@pytest.mark.parametrize( + ("label", "purpose"), + [ + ("Site_Builder / Build.sh", "site-builder-build-sh"), + ("../TAG:@!", "tag"), + ("", "template"), + ("💡", "template"), + ("x" * 200, "x" * 95), + ], +) +def test_image_tags_are_readable_valid_and_unique(label, purpose): + first = derive_image_name(label=label, repository="ghcr.io/acme/templates") + repeated = derive_image_name(label=label, repository="ghcr.io/acme/templates") + + assert first != repeated + assert first.startswith(f"ghcr.io/acme/templates:{purpose}-") + tag = first.rpartition(":")[2] + assert len(tag) <= 128 + assert re.fullmatch(r"[a-z0-9][a-z0-9-]*", tag) + + +async def test_delete_pinned_build_removes_its_tag(): + docker = MagicMock(remove_image=AsyncMock()) + tag = "ghcr.io/acme/templates:site-builder-build-unique" + + await remove_derived_image(docker, tag + "@sha256:" + "a" * 64) - assert first == repeated - assert first.startswith("drukbox-template:") - assert len(first.removeprefix("drukbox-template:")) == 12 - assert different_base != first + docker.remove_image.assert_awaited_once_with(tag) diff --git a/src/providers/docker/tests/test_provider.py b/src/providers/docker/tests/test_provider.py index 150006b..788ef4f 100644 --- a/src/providers/docker/tests/test_provider.py +++ b/src/providers/docker/tests/test_provider.py @@ -147,7 +147,7 @@ async def test_build_template_image_builds_and_returns_the_derived_tag(): ) assert image.startswith("drukbox-template:") - assert len(image.removeprefix("drukbox-template:")) == 12 + assert image.startswith("drukbox-template:node-tools-") assert api.build_image.await_args.args[0] == image diff --git a/src/providers/docker_sbx/provider.py b/src/providers/docker_sbx/provider.py index 655a5f1..49e8779 100644 --- a/src/providers/docker_sbx/provider.py +++ b/src/providers/docker_sbx/provider.py @@ -190,6 +190,7 @@ async def build_template_image( self.docker, base_image=base_image, setup_script=setup_script, + label=label, ) async def delete_template_image(self, image: str) -> None: diff --git a/src/providers/exe/provider.py b/src/providers/exe/provider.py index 3bbb8c7..9307d27 100644 --- a/src/providers/exe/provider.py +++ b/src/providers/exe/provider.py @@ -4,7 +4,6 @@ from providers.base import VMCreateResult, VMProvider from providers.capabilities import SecretInjectionCapability, TemplateCapability from providers.docker.api import DockerAPI -from providers.docker.exceptions import DockerProviderError from providers.docker.images import build_derived_image, remove_derived_image from providers.exceptions import ( ProviderCommandError, @@ -12,7 +11,6 @@ ProviderHttpProxyNotFoundError, ProviderNotFoundError, ProviderTargetVMNotFoundError, - ProviderTransportError, ) from providers.exe.api import ExeAPI from providers.exe.exceptions import ( @@ -68,19 +66,11 @@ async def create_vm( instance_type: str | None = None, disk_gb: int | None = None, ) -> VMCreateResult: - # exe.dev assumes a public image. A template pushed to the configured - # registry is private, so its pull gets --registry-auth (see - # https://exe.dev/docs/private-image). The credentials go only to - # the registry that they belong to. registry_auth = None - registry = self.settings.image_registry - if ( - registry - and self.settings.registry_username - and self.settings.registry_password - and image.partition("/")[0] == registry.partition("/")[0] - ): - registry_auth = f"{self.settings.registry_username}:{self.settings.registry_password}" + settings = get_settings() + if settings.registry_host and image.partition("/")[0] == settings.registry_host: + password = settings.registry_password.get_secret_value() + registry_auth = f"{settings.registry_username}:{password}" # Tags are operator-facing: `exe ls --tag=managed-by-` shows what this deployment owns. payload = await self.api.create_vm( @@ -115,35 +105,17 @@ async def build_template_image( setup_script: str, label: str, ) -> str: - registry = self.settings.image_registry - username = self.settings.registry_username - password = self.settings.registry_password - - if not (registry and username and password): - missing_settings = [ - name - for name, value in ( - ("EXE_IMAGE_REGISTRY", registry), - ("EXE_REGISTRY_USERNAME", username), - ("EXE_REGISTRY_PASSWORD", password), - ) - if not value - ] + if not get_settings().template_repository: raise ProviderCommandError( - f"exe template registry is not configured. Set: {', '.join(missing_settings)}" + "Template destination is not configured. Set TEMPLATE_REPOSITORY. " + "Configure REGISTRY_HOST, REGISTRY_USERNAME, and REGISTRY_PASSWORD for access." ) - - image = await build_derived_image( + return await build_derived_image( self.docker, base_image=base_image, setup_script=setup_script, - repository=registry, + label=label, ) - try: - await self.docker.push_image(image, username=username, password=password) - except DockerProviderError as exc: - raise ProviderTransportError(str(exc)) from exc - return image async def delete_template_image(self, image: str) -> None: # Registry deletion is registry-specific. This provider only removes diff --git a/src/providers/exe/settings.py b/src/providers/exe/settings.py index 602694f..7572654 100644 --- a/src/providers/exe/settings.py +++ b/src/providers/exe/settings.py @@ -22,21 +22,6 @@ class ExeSettings(BaseSettings): default_image: str = Field( description="Default VM image passed to exe.dev when provisioning.", ) - image_registry: str | None = Field( - default=None, - description=( - "Repository prefix for derived template images. A VM created from " - "this registry gets --registry-auth so exe.dev can pull a private image." - ), - ) - registry_username: str | None = Field( - default=None, - description="Username for the derived-template image registry.", - ) - registry_password: str | None = Field( - default=None, - description="Password or token for the derived-template image registry.", - ) api_timeout: float = Field( default=30.0, description="Timeout in seconds for exe.dev API calls.", diff --git a/src/providers/exe/tests/test_provider.py b/src/providers/exe/tests/test_provider.py index d534ee9..4b62277 100644 --- a/src/providers/exe/tests/test_provider.py +++ b/src/providers/exe/tests/test_provider.py @@ -3,7 +3,9 @@ from unittest.mock import AsyncMock import pytest +from pydantic import SecretStr +from core.settings import get_settings from providers.docker.api import DockerAPI from providers.docker.exceptions import DockerImageNotFoundError, DockerTransportError from providers.exceptions import ProviderCommandError, ProviderNotFoundError, ProviderTransportError @@ -22,7 +24,7 @@ def _docker_mock() -> SimpleNamespace: return SimpleNamespace( build_image=AsyncMock(), remove_image=AsyncMock(), - push_image=AsyncMock(), + push_image=AsyncMock(return_value="ghcr.io/acme/templates@sha256:" + "a" * 64), aclose=AsyncMock(), ) @@ -69,30 +71,41 @@ def _vm_payload() -> dict[str, str]: return {"vm_name": "sb-1", "ssh_port": "22", "ssh_dest": "sb-1.public.exe.dev"} -async def test_create_vm_sends_registry_auth_for_configured_registry_images() -> None: +@pytest.fixture +def registry_settings(monkeypatch): + settings = get_settings() + monkeypatch.setattr(settings, "registry_host", "ghcr.io") + monkeypatch.setattr(settings, "template_repository", "") + monkeypatch.setattr(settings, "registry_username", "bot") + monkeypatch.setattr(settings, "registry_password", SecretStr("secret")) + + +@pytest.mark.parametrize("suffix", [":abc123", "@sha256:" + "a" * 64, ":build@sha256:" + "a" * 64]) +async def test_create_vm_sends_registry_auth_without_template_configuration( + registry_settings, suffix +): api = SimpleNamespace(create_vm=AsyncMock(return_value=_vm_payload())) - settings = _settings( - image_registry="ghcr.io/acme/templates", - registry_username="bot", - registry_password="secret", - ) - provider = ExeProvider(api, settings, docker=_docker_mock()) # type: ignore[arg-type] + provider = _make_provider(api) - await provider.create_vm(name="sb-1", image="ghcr.io/acme/templates:abc123") + await provider.create_vm(name="sb-1", image=f"ghcr.io/acme/private-base{suffix}") assert api.create_vm.await_args.kwargs["registry_auth"] == "bot:secret" -async def test_create_vm_keeps_credentials_off_other_registries() -> None: +@pytest.mark.parametrize( + "image", + [ + "docker.io/library/ubuntu:24.04", + "ghcr.io.evil.example/acme/templates:tag", + "other.example/acme/templates:tag", + "ghcr.io:5000/acme/templates:tag", + ], +) +async def test_create_vm_keeps_credentials_off_other_registry_hosts(registry_settings, image): api = SimpleNamespace(create_vm=AsyncMock(return_value=_vm_payload())) - settings = _settings( - image_registry="ghcr.io/acme/templates", - registry_username="bot", - registry_password="secret", - ) - provider = ExeProvider(api, settings, docker=_docker_mock()) # type: ignore[arg-type] + provider = _make_provider(api) - await provider.create_vm(name="sb-1", image="docker.io/library/ubuntu:24.04") + await provider.create_vm(name="sb-1", image=image) assert api.create_vm.await_args.kwargs["registry_auth"] is None @@ -252,15 +265,14 @@ def test_from_settings_constructs_with_exeapi() -> None: assert isinstance(provider.docker, DockerAPI) -async def test_build_template_image_builds_logs_in_and_pushes() -> None: +async def test_build_template_image_builds_and_returns_digest( + registry_settings, monkeypatch +) -> None: + monkeypatch.setattr(get_settings(), "template_repository", "acme/templates") docker = _docker_mock() provider = ExeProvider( SimpleNamespace(), # type: ignore[arg-type] - _settings( - image_registry="ghcr.io/acme/drukbox-templates", - registry_username="builder", - registry_password="registry-secret", - ), + _settings(), docker=docker, # type: ignore[arg-type] ) @@ -270,12 +282,10 @@ async def test_build_template_image_builds_logs_in_and_pushes() -> None: label="Node tools", ) - assert image.startswith("ghcr.io/acme/drukbox-templates:") - assert len(image.rpartition(":")[2]) == 12 - assert docker.build_image.await_args.args[0] == image - docker.push_image.assert_awaited_once_with( - image, username="builder", password="registry-secret" - ) + assert image == "ghcr.io/acme/templates@sha256:" + "a" * 64 + tag = docker.build_image.await_args.args[0] + assert tag.startswith("ghcr.io/acme/templates:node-tools-") + docker.push_image.assert_awaited_once_with(tag, username="bot", password="secret") async def test_build_template_image_names_each_missing_registry_setting() -> None: @@ -293,22 +303,19 @@ async def test_build_template_image_names_each_missing_registry_setting() -> Non label="Node tools", ) - assert "EXE_IMAGE_REGISTRY" in str(error.value) - assert "EXE_REGISTRY_USERNAME" in str(error.value) - assert "EXE_REGISTRY_PASSWORD" in str(error.value) + assert "TEMPLATE_REPOSITORY" in str(error.value) + assert "REGISTRY_USERNAME" in str(error.value) + assert "REGISTRY_PASSWORD" in str(error.value) docker.build_image.assert_not_awaited() -async def test_build_template_image_translates_push_failure() -> None: +async def test_build_template_image_translates_push_failure(registry_settings, monkeypatch) -> None: + monkeypatch.setattr(get_settings(), "template_repository", "acme/templates") docker = _docker_mock() docker.push_image.side_effect = DockerTransportError("push log tail") provider = ExeProvider( SimpleNamespace(), # type: ignore[arg-type] - _settings( - image_registry="ghcr.io/acme/drukbox-templates", - registry_username="builder", - registry_password="registry-secret", - ), + _settings(), docker=docker, # type: ignore[arg-type] ) diff --git a/src/templates/tests/conftest.py b/src/templates/tests/conftest.py index 0304ef3..9e4632f 100644 --- a/src/templates/tests/conftest.py +++ b/src/templates/tests/conftest.py @@ -5,7 +5,6 @@ from providers import registry as registry_module from providers.base import VMCreateResult, VMProvider from providers.capabilities import TemplateCapability -from providers.docker.images import derive_image_name from providers.exceptions import ProviderError @@ -62,7 +61,7 @@ async def build_template_image( self.built.append((base_image, setup_script, label)) if self.build_error: raise self.build_error - return derive_image_name(base_image=base_image, setup_script=setup_script) + return "registry.example/templates@sha256:" + "a" * 64 async def delete_template_image(self, image: str) -> None: self.deleted.append(image) diff --git a/src/templates/tests/test_api.py b/src/templates/tests/test_api.py index 3a585bc..5aa549b 100644 --- a/src/templates/tests/test_api.py +++ b/src/templates/tests/test_api.py @@ -12,7 +12,6 @@ from core.database import async_session_factory from providers import registry as registry_module from providers.base import VMProvider -from providers.docker.images import derive_image_name from providers.exceptions import ProviderNotFoundError, ProviderTransportError from templates.exceptions import TemplateStateError from templates.models import Template, TemplateStatus @@ -51,10 +50,7 @@ async def test_create_template_returns_building_then_becomes_available(client, t assert polled.status_code == 200 assert polled.json()["status"] == TemplateStatus.AVAILABLE.value - assert polled.json()["image"] == derive_image_name( - base_image=template_provider.default_image, - setup_script=SETUP_SCRIPT, - ) + assert polled.json()["image"] == "registry.example/templates@sha256:" + "a" * 64 assert "setup_script" not in polled.json() assert template_provider.built == [ (template_provider.default_image, SETUP_SCRIPT, "Node tools")