Replies: 1 comment 1 reply
|
Hi, I don't think Testcontainers is the problem here. It doesn't download images on its own. It just asks Docker to pull the image, the same way docker pull does. So it isn't doing anything more aggressive than a normal pull. "Unknown blob" usually means the registry (in your case Artifactory) said "I don't have this file." So Artifactory is probably telling your workers that part of the image is missing, even though the image list says it should be there. That would also explain why pulling from your dev machine works: it may just have hit Artifactory at a good moment. Here's what I'd try: Pull by hand. The next time a build fails on a worker, run docker pull artifactory.local/confluentinc/cp-kafka:7.5.2 on that same machine. If it fails the same way, Testcontainers is ruled out. Until you find the cause, two things can help. You can add a small nightly job that pulls your common images (postgres, kafka, ryuk) on every worker. Testcontainers skips the pull if the image is already there, so your builds won't hit the problem, and different projects can just add their images to the list. You can also add a retry to the Jenkins test stage. If you post again, please include your docker version, docker info, your Artifactory version, and the type of repo (local, remote, or virtual). That will help people narrow it down. Hope this helps! |
|
Thanks for the pointers, we'll continue our investigation and hopefully find the root cause.
We've already tried that, |
Uh oh!
There was an error while loading. Please reload this page.
Hi,
We're experiencing the following error quite frequently on our CI servers since last month (full stack trace at the end of this post):
Context: we have a Jenkins infrastructure consisting of multiple workers. Each worker can run concurrent builds in "docker-in-docker" mode, meaning Jenkins pipelines spin their own Docker container to run maven builds. These builds are run using
-T1Cto build maven child modules in parallel, so in theory there's a chance that testcontainers might download the same image from multiple builds and/or multiple maven child modules at the same time.A
docker system prunewill usually remove broken layers and force a re-download during the next build, but after a few hours/days the problem comes back again and again. It happens on multiple images (ryuk, kafka, postgresql…), and images in different versions (postgresql 16.4, 16.8). It will eventually happen on any of our 8 workers.Pulling these images from one of our development machine always succeeds, suggesting that the corruption is not in our internal Artifactory instance, only in local caches in Jenkins workers.
From what I understand, a standard "docker pull" prevents cache corruption when run multiple times in parallel. Copilot and Gemini both seem to suggest that testcontainers works differently and is more "aggressive" when pulling images. It also states that we should rather use
docker pullcommands at the beginning of our build before starting testcontainers, which is not ideal since different projects will use different images in different versions.I can't really find anything regarding unknown blobs and testcontainers using a google search. I'm wondering if this kind of problems rings a bell for anyone in this community. Help would be really appreciated :)
Full stack trace:
All reactions