From 5980e41dc4657db95a1bae3e8d564b62fd0dbab6 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Tue, 4 Aug 2026 16:18:08 +0200 Subject: [PATCH 01/16] Fixed empty $my_dir outside the centos-7 branch be57fc62 computed $my_dir at the top of the script, and 1f25dbce relied on it for the linux-install-protobuf.sh and linux-install-rust.sh calls. dbc1efbc then moved the computation into the centos-7 branch so hosts without realpath would not abort under set -e, which left $my_dir empty for those two calls everywhere else. Resolved it once at the top again, calling realpath only when available so those hosts still get through. The fail-fast is gone with it: a relative dirname is enough to source a neighbouring script. Signed-off-by: Lars Erik Wik --- ci/fix-buildhost.sh | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/ci/fix-buildhost.sh b/ci/fix-buildhost.sh index d825bfc04..06d1ebd68 100755 --- a/ci/fix-buildhost.sh +++ b/ci/fix-buildhost.sh @@ -2,17 +2,19 @@ # it is expected that this file is sourced, not executed directly set -ex +# Resolve our own directory up front: the ci/ scripts sourced and run below live +# next to this file. Previously this was only computed in the centos-7 branch, +# leaving $my_dir empty for the linux-install-* calls further down. This file is +# sourced rather than executed, so BASH_SOURCE names it where $0 names the caller. +my_dir="$(dirname "${BASH_SOURCE[0]}")" +if command -v realpath >/dev/null; then + my_dir="$(realpath "$my_dir")" +fi + if [ -f /etc/os-release ]; then source /etc/os-release if [ "$ID" = "centos" ] && [ "$VERSION_ID" = "7" ]; then - if command -v realpath >/dev/null; then - my_path="$(realpath "${BASH_SOURCE[0]}")" - my_dir="$(dirname "$my_path")" - source "$my_dir"/centos-7-setup-devtoolset-11.sh - else - echo "FAIL: could not find realpath command on rhel/centos-7 to source needed centos-7-setup-devtoolset-11.sh" - exit 1 - fi + source "$my_dir"/centos-7-setup-devtoolset-11.sh fi fi From 9beba3a7635e94ebe9b5eedef10b1b60fcd2b194 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Tue, 4 Aug 2026 16:32:07 +0200 Subject: [PATCH 02/16] Moved the pinned SSH host keys into ci/known_hosts fix-buildhost.sh had the cache host key inline, and the containerized build (ENT-14361) needs the same keys. setup-ci-host.sh now seeds them too: jobs check their sources out over ssh, and cfengine-build-host-setup.cf, which writes that file today, is never reached from there. Also pins all three key types for the cache host rather than only ed25519, since which one is used is the client's choice. Ticket: ENT-14330 Signed-off-by: Lars Erik Wik --- ci/fix-buildhost.sh | 4 +++- ci/known_hosts | 16 ++++++++++++++++ ci/setup-ci-host.sh | 6 ++++++ 3 files changed, 25 insertions(+), 1 deletion(-) create mode 100644 ci/known_hosts diff --git a/ci/fix-buildhost.sh b/ci/fix-buildhost.sh index 06d1ebd68..7349c3a26 100755 --- a/ci/fix-buildhost.sh +++ b/ci/fix-buildhost.sh @@ -32,7 +32,9 @@ if [ -f /etc/profile ]; then fi mkdir -p ~/.ssh -echo "build-artifacts-cache.cloud.cfengine.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGahpsY8Phk2+isBmuJQjjQVlh6BNL/Qetc14g26gowV" >> ~/.ssh/known_hosts +# Only the cache host is needed here; github.com is not contacted from a build +# host. Which key type gets used depends on the client, so pin all of them. +grep '^build-artifacts-cache' "$my_dir"/known_hosts >> ~/.ssh/known_hosts # /etc/profile can contain tricky things, on suse for example it includes a call to tty which will fail in CI # so only source /etc/profile where we absolutely need it. diff --git a/ci/known_hosts b/ci/known_hosts new file mode 100644 index 000000000..199c97d03 --- /dev/null +++ b/ci/known_hosts @@ -0,0 +1,16 @@ +# Pinned SSH host keys for the hosts CI connects to. Consumers grep out the +# host they need: github.com for source checkouts on an agent, and +# build-artifacts-cache for the dependency cache (used inside the build +# container and by ci/fix-buildhost.sh on the build hosts). +# +# All key types each host offers are listed, since which one is used depends on +# the client's HostKeyAlgorithms preference. +# +# ci/cfengine-build-host-setup.cf holds its own inline copy. That policy is being +# replaced by these scripts (ENT-14330), so it is not worth coupling to. +github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl +github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= +github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= +build-artifacts-cache.cloud.cfengine.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGahpsY8Phk2+isBmuJQjjQVlh6BNL/Qetc14g26gowV +build-artifacts-cache.cloud.cfengine.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdv55BmDmwEjN3izaL8intrSRJQglkv++TaQppopKCh5VkYuSpNj/W+x0vjxwL3+NNp4CAhOLYuGTFuUL8zz/H0kwWE06c3WjghpHS3NS1usiamVZN6uaJMGwDaq8fPB3WiiI/L5lLM8/ubXxfmta0UlVufHCiBH8pBQarAY5rdPDtITXzu56qIa3k9Ou7Si2r/1n37espzwsPxoBqVJg7BvzMV28MzxdmQpGP6puuPfi9tvYbtnF788le3jvGOc+3GOiwtXsv44y/PCJNhi7sFeUfxocuNbWXZ3x/UEfbN8IiUVZsUAuLQFeqr3pv4w28D+SvJHBMCFudygenmLc3th+typiFRmqam7UQif9Pa3e2FxX4ghTp1KNXBoCQluIk2j7wX9zXppSyUG6d7tPDzfu81lImuW34+bsZvYq+s+25vbAhSDdQe1lqG0Fdvvi+zbqrMYQuMfnInDrK52xNSZcfATWjudhmY6wiwdSS0XsBADmZsy3qf3ErdEabqQk= +build-artifacts-cache.cloud.cfengine.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIzU5+SoC4gbtV3Wfw4oB6oMs5RYKGFCiS0lVeN4XQlAM8UjvyUUSflytf/vQEANv1OJs5vicslRn/iPlrvF8Mk= diff --git a/ci/setup-ci-host.sh b/ci/setup-ci-host.sh index c88f5d7a3..49b2ded2d 100755 --- a/ci/setup-ci-host.sh +++ b/ci/setup-ci-host.sh @@ -185,6 +185,12 @@ if command -v coredumpctl >/dev/null; then fi fi +# Host keys for the ssh source checkouts. These come from +# cfengine-build-host-setup.cf today, which will be obsolete in ENT-14330. +mkdir -p /home/jenkins/.ssh +cat "$thisdir"/known_hosts >> /home/jenkins/.ssh/known_hosts +chown -R jenkins:jenkins /home/jenkins/.ssh + "$thisdir"/linux-install-jdk.sh # the script should skip if sufficient java is already installed # leech2 build toolchain host From ddc0c0d5b0c60414f62980a1e93506f26c61aec6 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Tue, 4 Aug 2026 17:30:39 +0200 Subject: [PATCH 03/16] Added a docker-host role to setup-ci-host.sh Hosts for the build-in-container job need Docker with BuildKit, which no build host has today. Ticket: ENT-14361 Signed-off-by: Lars Erik Wik --- ci/setup-cfengine-build-host.sh | 2 +- ci/setup-ci-host.sh | 72 ++++++++++++++++++++++++++++++--- 2 files changed, 68 insertions(+), 6 deletions(-) diff --git a/ci/setup-cfengine-build-host.sh b/ci/setup-cfengine-build-host.sh index ecf8edbbd..32d257c0a 100755 --- a/ci/setup-cfengine-build-host.sh +++ b/ci/setup-cfengine-build-host.sh @@ -165,7 +165,7 @@ if [ -f /etc/cfengine-bootstrap-pr-host.flag ]; then exit fi -if [ -f /etc/cfengine-containers-host.flag ]; then +if [ -f /etc/cfengine-containers-host.flag ] || [ -f /etc/cfengine-docker-host.flag ]; then "$thisdir"/setup-ci-host.sh exit fi diff --git a/ci/setup-ci-host.sh b/ci/setup-ci-host.sh index 49b2ded2d..49691493f 100755 --- a/ci/setup-ci-host.sh +++ b/ci/setup-ci-host.sh @@ -77,6 +77,48 @@ EOF fi fi +# Hosts for the build-in-container job (ENT-14361). They only run containers: +# the target platform comes from the image, so none of the native build +# toolchain below is wanted here. +if [ -f /etc/cfengine-docker-host.flag ]; then + case "$ID" in + debian | ubuntu) ;; + *) + echo "docker host setup supports debian and ubuntu, not $ID" + exit 1 + ;; + esac + + # Docker CE from upstream rather than the distribution's docker.io, since + # build-in-container.py passes --build-context and so needs BuildKit. + # Follows https://docs.docker.com/engine/install/ubuntu/ ("Install using the + # apt repository"); the debian page has the same steps with the other URI. + apt-get -y install ca-certificates curl + install -m 0755 -d /etc/apt/keyrings + curl -fsSL "https://download.docker.com/linux/$ID/gpg" -o /etc/apt/keyrings/docker.asc + chmod a+r /etc/apt/keyrings/docker.asc + tee /etc/apt/sources.list.d/docker.sources << EOF +Types: deb +URIs: https://download.docker.com/linux/$ID +Suites: ${UBUNTU_CODENAME:-$VERSION_CODENAME} +Components: stable +Architectures: $(dpkg --print-architecture) +Signed-By: /etc/apt/keyrings/docker.asc +EOF + apt-get -qy update + + # docker-compose-plugin, the fifth package the documented command installs, + # is deliberately left out: nothing we run calls docker compose. + add-pkg containerd.io + add-pkg docker-buildx-plugin + add-pkg docker-ce + add-pkg docker-ce-cli + add-pkg git # the pipeline checks the source repos out on the agent + add-pkg jq + add-pkg python3 # runs build-in-container.py + add-pkg rsync +fi + if [ "$redhat" != 0 ]; then if [ "$redhat" -gt 7 ]; then if ! grep best=False /etc/yum.conf; then @@ -193,11 +235,31 @@ chown -R jenkins:jenkins /home/jenkins/.ssh "$thisdir"/linux-install-jdk.sh # the script should skip if sufficient java is already installed -# leech2 build toolchain host -if [ "$ubuntu" -ge 20 ] || [ "$debian" -ge 12 ] || [ "$redhat" -ge 7 ]; then - "$thisdir"/linux-install-protobuf.sh - # TODO if mingw then pass along x86_64-pc-windows-gnu as an arg to install rust - "$thisdir"/linux-install-rust.sh +if [ -f /etc/cfengine-docker-host.flag ]; then + systemctl enable --now docker + + # Give jenkins access to the docker socket, per + # https://docs.docker.com/engine/install/linux-postinstall/. + groupadd -f docker + usermod -aG docker jenkins + + # Dependency cache root for build-in-container.py's --cache-dir. Outside any + # workspace so that cleanWs() cannot wipe it between builds. + install -d -o jenkins -g jenkins /home/jenkins/cfengine-build-cache + + docker --version + docker buildx version + sudo -u jenkins docker info +fi + +# leech2 build toolchain host. Not on a docker host, where the toolchain belongs +# in the build images. +if [ ! -f /etc/cfengine-docker-host.flag ]; then + if [ "$ubuntu" -ge 20 ] || [ "$debian" -ge 12 ] || [ "$redhat" -ge 7 ]; then + "$thisdir"/linux-install-protobuf.sh + # TODO if mingw then pass along x86_64-pc-windows-gnu as an arg to install rust + "$thisdir"/linux-install-rust.sh + fi fi if [ "$redhat" -ge 7 ]; then From 1b735d89a7d05777718c4c5ddd3b870a1a45e381 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Wed, 5 Aug 2026 13:03:43 +0200 Subject: [PATCH 04/16] Now uses same labels as testing-pr for buildcache Ticket: ENT-14362 Signed-off-by: Lars Erik Wik --- build-in-container.py | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/build-in-container.py b/build-in-container.py index 372b19385..40112e73f 100755 --- a/build-in-container.py +++ b/build-in-container.py @@ -344,6 +344,28 @@ def update_base_image_shas(platform_name=None): CONFIG_PATH.write_text(json.dumps(config, indent=2) + "\n") +def cache_label(platform_name, role, arch): + """Return the dependency cache namespace for a build. + + deps-packaging/pkg-cache namespaces cached dependencies by JOB_BASE_NAME, + which a testing-pr matrix cell exports as "label=". Building the + same string here puts container-built dependencies in the same namespace as + the ones testing-pr builds, so both jobs share buildcache. + """ + hub = "_HUB" if role == "hub" else "" + # The labels spell the architectures x86_64 and arm_64. See labels.txt. + arch_token = {"amd64": "x86_64", "arm64": "arm_64"}[arch.rsplit("/", 1)[-1]] + + # The cross target's label carries neither an OS version nor _linux. + if get_config()[platform_name].get("cross_target"): + return f"PACKAGES{hub}_{arch_token}_mingw" + + # Platform names are -, matching the labels once the separator + # is swapped, except that the labels say redhat where we say rhel. + label_os = platform_name.replace("-", "_").replace("rhel_", "redhat_") + return f"PACKAGES{hub}_{arch_token}_linux_{label_os}" + + def run_container(args, image_tag, source_dir, script_dir): """Run the build inside a Docker container.""" output_dir = Path(args.output_dir).resolve() @@ -376,7 +398,7 @@ def run_container(args, image_tag, source_dir, script_dir): # Environment variables # JOB_BASE_NAME is used by deps-packaging/pkg-cache to derive the cache # label. Format: "label=". Without it, all platforms share NO_LABEL. - cache_label = f"label=container_{args.platform}" + label = cache_label(args.platform, args.role, args.arch or host_docker_arch()) cmd.extend( [ "-e", @@ -388,7 +410,7 @@ def run_container(args, image_tag, source_dir, script_dir): "-e", f"BUILD_NUMBER={args.build_number}", "-e", - f"JOB_BASE_NAME={cache_label}", + f"JOB_BASE_NAME=label={label}", "-e", "CACHE_IS_ONLY_LOCAL=yes", ] From 3527db68364096b0b3743a0c3c61d90feb459092 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Wed, 5 Aug 2026 14:18:42 +0200 Subject: [PATCH 05/16] Stopped syncing stale revision files into the container autogen only writes a revision file when one is absent, and rsync brought the host's along, so a leftover from an earlier build decided the dependency cache key. Signed-off-by: Lars Erik Wik --- build-in-container-inner.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/build-in-container-inner.sh b/build-in-container-inner.sh index af39b0618..2c9a014ee 100755 --- a/build-in-container-inner.sh +++ b/build-in-container-inner.sh @@ -33,11 +33,15 @@ for repo in $repos; do # over from previous test runs and are not needed for building. # Also skip node_modules/vendor for hub builds. # Also skip compilation results *.o, *.lo, *.la as the local copy is likely a different platform/OS than inside the container + # Skip revision files too: autogen only writes them when absent, so a + # leftover from an earlier host build would key the dependency cache to + # whatever commit that build saw. if [ -d "$src" ] || [ -L "$src" ]; then echo "Syncing $repo..." sudo rsync -aL --exclude='config.cache' --exclude='workdir' \ --exclude='*.o' --exclude='*.lo' --exclude='*.la' \ --exclude='node_modules' --exclude='vendor' \ + --exclude='revision' \ --chown="$(id -u):$(id -g)" "$src/" "$BASEDIR/$repo/" else echo "ERROR: Required repository $repo not found" >&2 From 4ae5a281fa09a563b2522fd1063c8e7082f24b11 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Wed, 5 Aug 2026 14:21:30 +0200 Subject: [PATCH 06/16] Added --sftp-key to reach the remote dependency cache Ticket: ENT-14361 Signed-off-by: Lars Erik Wik --- build-in-container-inner.sh | 15 +++++++++++++++ build-in-container.md | 2 ++ build-in-container.py | 24 ++++++++++++++++++++++-- 3 files changed, 39 insertions(+), 2 deletions(-) diff --git a/build-in-container-inner.sh b/build-in-container-inner.sh index 2c9a014ee..7e384440d 100755 --- a/build-in-container-inner.sh +++ b/build-in-container-inner.sh @@ -49,6 +49,21 @@ for repo in $repos; do fi done +# The dependency cache is reached over sftp, so the key has to be in place +# before install-dependencies runs. It arrives on a read-only mount owned by the +# host user, and ssh refuses a key owned by anyone but us, hence the copy. +if [ -f /run/secrets/sftp-cache-key ]; then + echo "Installing dependency cache key..." + install -d -m 700 "$HOME/.ssh" + install -m 600 /run/secrets/sftp-cache-key "$HOME/.ssh/id_rsa" + grep '^build-artifacts-cache' "$BASEDIR/buildscripts/ci/known_hosts" \ + >> "$HOME/.ssh/known_hosts" + + # Fail now rather than once every dependency has been built, which is when + # pkg-cache would first try to upload. + echo pwd | sftp -o BatchMode=yes -b - jenkins_sftp_cache@build-artifacts-cache.cloud.cfengine.com +fi + # Pin embedded build timestamps so two builds of the same source produce # identical binaries. Honored by OpenSSL, Apache httpd, Postgres, Python # (.pyc mtimes), dpkg-buildpackage, and rpmbuild. diff --git a/build-in-container.md b/build-in-container.md index 778be7813..b6da8ca75 100644 --- a/build-in-container.md +++ b/build-in-container.md @@ -46,6 +46,7 @@ None of the above arguments are required for `--update`. | ------------------ | -------------------------------- | ---------------------------------------------------------------------------------- | | `--output-dir` | `./output` | Where to write output packages | | `--cache-dir` | `~/.cache/cfengine/buildscripts` | Dependency cache directory | +| `--sftp-key` | | Private key for the remote dependency cache. Uses local cache if not specified | | `--build-number` | `1` | Build number for package versioning | | `--version` | auto | Override version string | | `--rebuild-image` | | Force rebuild of Docker image (bypasses Docker layer cache) | @@ -169,6 +170,7 @@ The system has three components: | Source repos (parent of `buildscripts/`) | `/srv/source` | read-only | Protects host repos from modification | | `~/.cache/cfengine/buildscripts/` | `/home/builder/.cache/buildscripts_cache` | read-write | Dependency cache shared across builds | | `./output/` | `/output` | read-write | Output packages copied here | +| `--sftp-key` (when given) | `/run/secrets/sftp-cache-key` | read-only | Key for the remote dependency cache | ### Build steps diff --git a/build-in-container.py b/build-in-container.py index 40112e73f..88398f1fd 100755 --- a/build-in-container.py +++ b/build-in-container.py @@ -21,6 +21,12 @@ IMAGE_REGISTRY = "ghcr.io/cfengine" CONFIG_PATH = Path(__file__).resolve().parent / "platforms.json" +# Where --sftp-key is mounted. It cannot be mounted onto ~/.ssh/id_rsa directly: +# ssh rejects a key owned by neither the current user nor root, and the host file +# belongs to jenkins while the container runs as builder. The inner script copies +# it into place instead. +SFTP_KEY_PATH = "/run/secrets/sftp-cache-key" + # Architectures registry images are published for, unless a platform overrides # it with an "architectures" list in platforms.json (e.g. the mingw cross-build, # which always targets Windows x64 and only makes sense on amd64). @@ -411,11 +417,19 @@ def run_container(args, image_tag, source_dir, script_dir): f"BUILD_NUMBER={args.build_number}", "-e", f"JOB_BASE_NAME=label={label}", - "-e", - "CACHE_IS_ONLY_LOCAL=yes", ] ) + # The remote dependency cache is reachable by publickey only, and pkg-cache + # aborts the build if an upload fails, so it stays off unless a key was + # passed. Note that the key is readable by everything the build runs, + # including each dependency's own build system. + if args.sftp_key: + key = Path(args.sftp_key).resolve() + cmd.extend(["-v", f"{key}:{SFTP_KEY_PATH}:ro"]) + else: + cmd.extend(["-e", "CACHE_IS_ONLY_LOCAL=yes"]) + if args.version: cmd.extend(["-e", f"EXPLICIT_VERSION={args.version}"]) @@ -487,6 +501,12 @@ def parse_args(): default=str(Path.home() / ".cache" / "cfengine" / "buildscripts"), help="Dependency cache directory", ) + parser.add_argument( + "--sftp-key", + dest="sftp_key", + help="Private key for the remote dependency cache. Without it the build " + "only uses the local cache under --cache-dir.", + ) parser.add_argument( "--rebuild-image", action="store_true", From d4d6fd2fdce8501a3596fdd085a449e4ef5c9697 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Wed, 5 Aug 2026 14:49:12 +0200 Subject: [PATCH 07/16] build-in-container.md: made it more concise and less of a maintanance burden Signed-off-by: Lars Erik Wik --- build-in-container.md | 126 +++++++++++++----------------------------- 1 file changed, 38 insertions(+), 88 deletions(-) diff --git a/build-in-container.md b/build-in-container.md index b6da8ca75..013fe9ada 100644 --- a/build-in-container.md +++ b/build-in-container.md @@ -23,93 +23,45 @@ specified, defaults will: (`~/.cache/cfengine/buildscripts`). - Use the current working directory for output packages (`./output/`). -## Usage +### Usage +See: +```bash +$ ./build-in-container.py --help ``` -./build-in-container.py --platform PLATFORM --project PROJECT --role ROLE --build-type TYPE [OPTIONS] -``` - -### Required arguments - -| Option | Description | -| -------------- | ------------------------------------------------------- | -| `--platform` | Target platform (e.g. `ubuntu-22`, `debian-12`) | -| `--project` | `community` or `nova` (not required for `--push-image`) | -| `--role` | `agent` or `hub` (not required for `--push-image`) | -| `--build-type` | `DEBUG` or `RELEASE` (not required for `--push-image`) | - -None of the above arguments are required for `--update`. - -### Optional arguments - -| Option | Default | Description | -| ------------------ | -------------------------------- | ---------------------------------------------------------------------------------- | -| `--output-dir` | `./output` | Where to write output packages | -| `--cache-dir` | `~/.cache/cfengine/buildscripts` | Dependency cache directory | -| `--sftp-key` | | Private key for the remote dependency cache. Uses local cache if not specified | -| `--build-number` | `1` | Build number for package versioning | -| `--version` | auto | Override version string | -| `--rebuild-image` | | Force rebuild of Docker image (bypasses Docker layer cache) | -| `--push-image` | | Build image and push to registry, then exit | -| `--update` | | Fetch latest image versions from registry and update platforms.json | -| `--update-sha` | | Fetch latest base image manifest digests from Docker Hub and update platforms.json | -| `--shell` | | Drop into a bash shell inside the container for debugging | -| `--list-platforms` | | List available platforms and exit | -| `--source-dir` | parent of `buildscripts/` | Root directory containing repos | -| `--arch` | host architecture | Override the container architecture (see [Architecture](#architecture)) | ## Supported platforms -| Name | Base image | -| ----------- | -------------------------- | -| `ubuntu-20` | `ubuntu:20.04` | -| `ubuntu-22` | `ubuntu:22.04` | -| `ubuntu-24` | `ubuntu:24.04` | -| `debian-11` | `debian:11` | -| `debian-12` | `debian:12` | -| `debian-13` | `debian:13` | -| `rhel-8` | `rockylinux/rockylinux:8` | -| `rhel-9` | `rockylinux/rockylinux:9` | -| `rhel-10` | `rockylinux/rockylinux:10` | - -RHEL packages are built on Rocky Linux base images. The build scripts detect -`OS=rhel` from `/etc/redhat-release` (which reports `Rocky Linux release ...`), -so the produced `.rpm`s are ordinary Red Hat / rpm packages. AlmaLinux is _not_ -recognized by `build-scripts/detect-environment`, which is why Rocky is used. - -Adding a new Debian/Ubuntu platform requires a new entry in `platforms.json` -and adding the platform name to the matrix in +See: +```bash +$ ./build-in-container.py --list-platforms +``` + +Adding a new platform normally requires a new entry in `platforms.json` and +adding the platform name to the matrix in `.github/workflows/build-base-images.yml` so the weekly job builds and -pushes its image to `ghcr.io`. Without the matrix entry, no image is ever -pushed and the `update-base-images.yml` workflow will fail with a 403 from -`ghcr.io` when it queries tags for the missing repository. +pushes its image to `ghcr.io`. The new entry in `platforms.json` needs: - `image_version`: set to `"latest"` as a placeholder. The - `update-base-images.yml` workflow (or `./build-in-container.py --update` - run locally) will replace it with the real ghcr.io tag after the first - image is pushed. -- `base_image_sha`: the Docker Hub manifest digest for the `base_image`. - Don't copy this by hand — run `./build-in-container.py --update-sha ---platform ` and it will fetch the current digest from - Docker Hub and write it into `platforms.json`. -- `architectures` (optional): the list of docker platforms to publish, e.g. - `["linux/amd64", "linux/arm64"]`. Omit it to get the multi-arch default; set - it only to restrict a platform to specific architectures (see - [Architecture](#architecture)). + `update-base-images.yml` workflow will replace it with the real ghcr.io tag + after the first image is pushed. +- `base_image_sha`: the Docker Hub manifest digest for the `base_image`. Don't + copy this by hand -- run + `./build-in-container.py --update-sha --platform ` and it will + fetch the current digest from Docker Hub and write it into `platforms.json`. -Adding another RHEL-family platform (a new Rocky/RHEL major version) works the -same way: add a `platforms.json` entry with `"dockerfile": "Dockerfile.rhel"` -and a matrix entry, then set any per-version `extra_build_args` — `CRB_REPO` -(`powertools` on 8, `crb` on 9+), `PHP_MODULE_STREAM` (`remi-8.3` where the -distro's default PHP is older than 8.3; RHEL 10 already ships 8.3), and -`EXTRA_PKGS` for version-specific packages. Note that `--update-sha` also works -for the namespaced `rockylinux/rockylinux` base images, not just official -Docker Hub library images. +Optionally, add the following entries: + +- `architectures`: the list of docker platforms to publish, e.g. + `["linux/amd64", "linux/arm64"]`. Omit it to get the multi-arch default; set + it only to restrict or extend the architectures. +- `extra_build_args`: allows you to add extra arguments through environment + variables. -Adding an entirely different, non-RHEL/non-Debian platform family (e.g. SUSE) -would require a new `container/Dockerfile.` plus platform entries. +Adding an entirely different platform family (e.g. SUSE) would require a new +`container/Dockerfile.`. ## Architecture @@ -128,7 +80,7 @@ The registry images are published as multi-arch manifests (`linux/amd64` and `linux/arm64`), so `--arch` normally just pulls the matching variant. If the registry does not provide the requested architecture (for example an older, single-arch image that predates multi-arch support), the script falls back to -building the image locally for that architecture. +building the image locally for that architecture only. Building a non-host architecture - whether locally or in CI - relies on QEMU/binfmt emulation being registered on the build host. If it isn't set up, @@ -138,7 +90,7 @@ register it once with: docker run --privileged --rm tonistiigi/binfmt --install all ``` -Emulated builds are considerably slower than native ones. +Please note that emulated builds are considerably slower than native ones. The set of architectures published for each platform defaults to `linux/amd64` and `linux/arm64`. A platform can override this with an `"architectures"` list @@ -150,18 +102,16 @@ is pinned to `["linux/amd64"]`. The system has three components: -1. **`build-in-container.py`** (Python) -- the orchestrator that runs on the host. - Parses arguments, builds the Docker image, and launches the container with - the correct mounts and environment variables. +1. **`build-in-container.py`** (Python) -- the orchestrator that runs on the + host. Parses arguments, builds the Docker image, and launches the container + with the correct mounts and environment variables. 2. **`build-in-container-inner.sh`** (Bash) -- runs inside the container. Copies - source repos from the read-only mount, then calls the existing build scripts - in order. + source repos from the read-only mount, then calls the build scripts in order. -3. **`container/Dockerfile.debian`** and **`container/Dockerfile.rhel`** -- - parameterized Dockerfiles shared across platforms of the same family via a - `BASE_IMAGE` build arg (plus per-platform `extra_build_args` in - `platforms.json`, e.g. the CRB repo name and PHP module stream for RHEL). +3. **`container/Dockerfile.`** -- parameterized Dockerfiles shared + across platforms of the same family via a `BASE_IMAGE` build arg (plus + per-platform `extra_build_args` in `platforms.json`. ### Container mounts @@ -241,8 +191,8 @@ specific Docker Hub manifest. To refresh them to the current digests: ./build-in-container.py --update-sha --platform ubuntu-22 ``` -The `update-base-image-shas.yml` workflow automates this. It runs weekly -(Monday at 01:00 UTC) and opens a pull request with any digest changes. +The `update-base-image-shas.yml` workflow automates this. It runs weekly (Monday +at 01:00 UTC) and opens a pull request with any digest changes. The workflow authenticates to `ghcr.io` using the automatic `GITHUB_TOKEN` provided by GitHub Actions. For this to work: From 22deeb913243f428c95ecb3ad7b3195b5795470d Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Wed, 5 Aug 2026 15:03:54 +0200 Subject: [PATCH 08/16] Put built packages in a directory named after the label Building several platforms into one output directory mixed their packages together. Keep them apart the way testing-pr does, which copies each cell's artifacts to upload/$label. The directory is emptied first, since the label says nothing about the project or the build type, and yesterday's packages should not look like this build's output. Ticket: ENT-14361 Signed-off-by: Lars Erik Wik --- build-in-container.md | 2 +- build-in-container.py | 19 ++++++++++++++----- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/build-in-container.md b/build-in-container.md index 013fe9ada..4759825f0 100644 --- a/build-in-container.md +++ b/build-in-container.md @@ -119,7 +119,7 @@ The system has three components: | ---------------------------------------- | ----------------------------------------- | ---------- | ------------------------------------- | | Source repos (parent of `buildscripts/`) | `/srv/source` | read-only | Protects host repos from modification | | `~/.cache/cfengine/buildscripts/` | `/home/builder/.cache/buildscripts_cache` | read-write | Dependency cache shared across builds | -| `./output/` | `/output` | read-write | Output packages copied here | +| `./output/