Skip to content

[deckhouse-cli] Fix extra images dropped on network errors#418

Draft
Glitchy-Sheep wants to merge 10 commits into
mainfrom
fix/skipping-extra-images-during-network-errors
Draft

[deckhouse-cli] Fix extra images dropped on network errors#418
Glitchy-Sheep wants to merge 10 commits into
mainfrom
fix/skipping-extra-images-during-network-errors

Conversation

@Glitchy-Sheep

@Glitchy-Sheep Glitchy-Sheep commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

During module pull, extra images are discovered by reading extra_images.json from each version image. A network error while reading it was treated the same as "this version has no extra images", so the images were dropped and the pull still reported success.

Problem

  • findExtraImages swallowed every error from GetImage and from reading extra_images.json.
  • A network error looked the same as "this version has no extra_images.json".
  • img.Extract() made it worse: on a mid-layer read failure it returns a clean io.EOF, so even the network error itself was indistinguishable from an absent file.
  • The dropped image never entered the download plan. The pull's retry and completeness check never saw it.
  • The bundle ended up missing extra images, with no error shown to the user.

Fix

  • Read extra_images.json by scanning the image layers directly instead of img.Extract(), so a truncated or failed layer read surfaces as a real error rather than a masked io.EOF.
  • Read each version's extra_images.json through the shared retry.RunTask primitive (5 attempts, 10s apart) - the same retry the image-pull step already uses.
  • A missing image or a missing extra_images.json is a clean skip.
  • A transient error is retried. A persistent one is returned.
  • findExtraImages now returns an error and pullExtraImages propagates it up.
  • A persistent failure now fails the whole pull instead of writing an incomplete bundle.

Before / After

Before: a network error during extra images discovery dropped that version's extra images, and the pull still finished successfully.

After: the read is retried; if it keeps failing, the pull stops with a clear error instead of producing an incomplete bundle.

2026-07-26 AT 01 48@2x

Tests

  • TestFindExtraImages_Discovery - a version with extra_images.json yields its extra images; a version without it, or with a missing image, is skipped.
  • TestFindExtraImages_RetriesTransientError - a transient GetImage error is retried and the extra images still land.
  • TestFindExtraImages_FailsOnPersistentError - a persistent error returns an error instead of an empty result.
  • TestExtractExtraImagesJSON_LayerReadErrorIsNotSkipped - a layer read that aborts mid-stream returns a real error, not a masked "not found".

Notes

  • Scope is limited to the modules extra-images path. The same swallow-and-mask pattern exists in extractInternalDigestImages and extractVersionsFromReleaseChannels, whose pull step is best-effort, so they are left out on purpose.
  • packages/packages.go findExtraImages is a near-duplicate of this code and has the same bug; a follow-up should apply the same fix there.

- `errExtraImagesJSONNotFound` lets callers tell an absent `extra_images.json` from a network read error via `errors.Is`.
- Groundwork so the discovery step can retry transient failures instead of skipping the module version.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- `getExtraImagesJSON` retries transient registry errors when
  reading a module version's extra_images.json.
- A missing image or missing extra_images.json returns (nil, nil):
  a version without extra images is expected, not a failure.
- A persistent error is returned so the caller can fail the pull
  instead of quietly dropping extra images.
- Retries are logged via `userLogger.Warnf`, visible without debug.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- `findExtraImages` reads each version through `getExtraImagesJSON`, so a transient registry error is retried instead of dropping the version's extra images.
- A persistent error propagates through `pullExtraImages` and fails the pull, rather than getting swallowed at debug level while the bundle ends up missing extra images.
- A version with no extra_images.json is still skipped cleanly.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- Cover `findExtraImages`: extra_images.json is parsed into extra image refs, and a version without it or with a missing image is skipped.
- A transient GetImage error is retried and the extra images still land once the registry recovers.
- A persistent error fails discovery instead of silently dropping the version's extra images.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep Glitchy-Sheep self-assigned this Jul 24, 2026
@Glitchy-Sheep Glitchy-Sheep added the bug Something isn't working label Jul 24, 2026
@Glitchy-Sheep Glitchy-Sheep changed the title [deckhouse-cli] Add sentinel error for missing extra_images.json [deckhouse-cli] Fix extra images dropped on network errors Jul 24, 2026
- `mutate.Extract` masks a failed mid-layer read as a clean io.EOF, so a network error looked identical to extra_images.json being absent.
- The version's extra images were then dropped silently, which the earlier retry/fail-loud handling never got a chance to catch.
- Scanning the image layers directly surfaces a truncated read as a real error.
- The caller retries that error and fails the pull when it persists, instead of skipping the version.
- `errExtraImagesJSONNotFound` is returned only when a layer is read in full without the file present.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- Cover extractExtraImagesJSON directly: a layer read that aborts
  mid-stream returns a real error, not errExtraImagesJSONNotFound.
- This pins the fix for the masked-EOF bug so it can't silently regress.
- Keep the happy path and the genuinely-absent case: a present file
  is parsed, a fully-read layer without the file is a clean not-found.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- packages/packages.go had its own copy of extractExtraImagesJSON
  that read via img.Extract(), which masks a mid-layer read failure
  as a clean io.EOF - indistinguishable from an absent file.
- Scan the image layers directly so a truncated or failed read
  surfaces as a real error, mirroring the modules fix.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- getExtraImagesJSON reads a package version's extra_images.json
  through the shared retry.RunTask primitive, retrying transient
  registry errors like the modules path now does.
- A missing image or missing extra_images.json returns (nil, nil):
  a version without extra images is expected, not a failure.
- A persistent error is returned so the caller can fail the pull
  instead of quietly dropping extra images.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- findExtraImages reads each version via getExtraImagesJSON, so a
  transient registry error is retried instead of dropping the
  version's extra images.
- A persistent error now propagates through pullExtraImages and
  fails the pull, instead of being swallowed at debug level and
  producing a bundle silently missing extra images.
- A version with no extra_images.json is still skipped cleanly.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- Cover packages findExtraImages/extractExtraImagesJSON, mirroring
  the modules tests: present/absent/not-found, transient retry,
  persistent failure, and a masked layer-read error.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant