[BUG] 0.9.0 regression: GPU JPEG ROI decode via nvimgcodecCodeStreamGetSubCodeStream fails on ordinary in-bounds ROIs — INTERNAL_ERROR (8) on some GPUs, SIGSEGV on others (worked in 0.8.0)

[janbernloehr]

Version

0.9.0 — specifically the nvidia-nvimgcodec-cu13 0.9.0.1859 wheel.

Last known good: nvidia-nvimgcodec-cu13 0.8.0.1422.

Describe the bug.

GPU (device="mixed") JPEG region-of-interest decoding regressed in 0.9.0.

DALI 2.3.0 implements fn.decoders.image_random_crop by creating a sub-code-stream with a region view — nvimgcodecCodeStreamGetSubCodeStream(parent, &sub, &cs_view) where cs_view.image_idx = 0 and cs_view.region is an nvimgcodecRegion_t with ndim = 2 and start[]/end[] in raw codestream pixel coordinates — and then decodes that sub-stream on the GPU. Against 0.9.0.1859 this fails on the very first batch for ordinary, in-bounds ROIs on standard ImageNet-1k JPEG training images. Against 0.8.0.1422 the identical pipeline runs normally.

There are two failure modes, and which one occurs correlates with the GPU model, not with the request:

GPU	Observed
A100 80GB, GB200 NVL, GB300 NVL	decode fails; DALI surfaces nvImageCodec failure: '#8' (NVIMGCODEC_STATUS_INTERNAL_ERROR)
H100 80GB, B200	process terminates with SIGSEGV (exit 139, core dumped)

The same recipe, same image set, same DALI version, and same nvImageCodec wheel produce the error on one GPU model and a hard crash on another. A segmentation fault on a well-formed public API call is the more serious half of this report: even for a malformed request the library should return a status rather than corrupt memory.

Expected behavior: the requested region is decoded, as in 0.8.0.1422.
Actual behavior: NVIMGCODEC_STATUS_INTERNAL_ERROR, or SIGSEGV, depending on GPU model.

This is not EXIF-orientation related

We initially suspected the interaction between EXIF orientation and ROI coordinate spaces, and tested it directly with a controlled A/B: the only changed variable was DALI's adjust_orientation argument on the decoder (which sets nvimgcodecDecodeParams_t.apply_exif_orientation), with container image, GPU, driver, image set, batch size and all other pipeline parameters held fixed.

Both arms failed identically. With adjust_orientation=False — i.e. apply_exif_orientation = 0, no orientation-driven shape transposition on DALI's side, and a region expressed in raw codestream coordinates derived from the raw JPEG shape — the A100 run still returned #8 and the H100 run still segfaulted, at the same point as the corresponding baseline runs. Disabling orientation handling entirely does not avoid the defect, so the failing case is a plain in-bounds ROI decode.

Where we suspect the problem is (pointers, not conclusions)

We root-caused as far as we can from outside the library and did not determine which internal mechanism is responsible. Two candidates are consistent with the evidence; both look like nvImageCodec-side defects, and the cheapest checks to separate them are listed under Other/Misc. below.

1. GPU-model-conditional ROI dispatch (would explain error-vs-crash). In extensions/nvjpeg/hw_decoder.cpp, chooseDecodePath() selects DecodePath::Legacy vs DecodePath::BatchSingle for a single-image decode based on num_hw_engines_ <= 1. A single-image ROI decode therefore runs structurally different ROI code depending on the GPU's HW JPEG engine count, which is exactly the axis along which we observe the split. In the BatchSingle device stage, the region is re-read through slot.codestream_info_.code_stream_view->region — a raw pointer into the owning CodeStream — long after the host stage; that carries a Coverity forward_null suppression and looks like the most plausible crash candidate, though we have not traced the lifetime end to end.
2. Deferred/absent region validation. validateCodeStreamViewSelection() (src/nvimgcodec_capi.cpp) does not inspect region at all — no ndim check, no start <= end, no bounds check against the parent image — and for a region-only view (image_idx == 0, bitstream_offset == 0) the eager code-stream-info resolution branch is skipped, so everything defers to decode time. Relatedly, ndim == 2 is enforced by assert() only in the decoder hot paths, which compiles out under NDEBUG.

We also noticed nvimgcodecCodeStreamView_t lost a trailing uint32_t limit_images between the 0.8.0 and 0.9.0 release drops, changing sizeof(nvimgcodecCodeStreamView_t). Since CHECK_STRUCT_SIZE raises INTERNAL_ERROR (the same status 8 we observe) on a size mismatch, a caller compiled against a 0.8 header but running against a 0.9 runtime would fail this way. We could not confirm whether that mix actually occurs in our builds, and it would not explain the GPU-dependent segfault — we mention it only so it can be ruled in or out quickly, and because removing a member from a public struct in a minor release with no layout guard, plus reporting the mismatch as INTERNAL_ERROR rather than INVALID_PARAMETER, seem worth reviewing regardless.

Minimum reproducible example

# Exercises the same operator and code path as the failing workload:
# a MIXED (GPU) JPEG ROI decode, which DALI implements via
# nvimgcodecCodeStreamGetSubCodeStream + a region view.
#
# Honest scoping note: the failure was observed and A/B-tested in a full
# ImageNet ResNet-50 / EfficientNet-V2-S training pipeline on the GPUs listed
# above. The snippet below is the reduction of that pipeline's decode stage;
# it has NOT itself been executed in isolation, because the environment used
# to prepare this report has no GPU. The decoder call is reproduced verbatim
# from the failing pipeline.

from nvidia.dali import pipeline_def, fn, types

@pipeline_def(batch_size=128, num_threads=10, device_id=0)
def training_pipe(data_dir):
    jpegs, labels = fn.readers.file(file_root=data_dir, random_shuffle=True)
    images = fn.decoders.image_random_crop(
        jpegs,
        device="mixed",
        output_type=types.RGB,
        random_aspect_ratio=[0.75, 4.0 / 3.0],
        random_area=[0.08, 1.0],
    )
    return images, labels

pipe = training_pipe(data_dir="/path/to/imagenet/train")
pipe.build()
pipe.run()   # fails on the first batch with nvImageCodec 0.9.0.1859

Run the same script against nvidia-nvimgcodec-cu13==0.8.0.1422 (with the matching DALI release) for the passing comparison.

The underlying C API sequence, per DALI's public source in dali/operators/imgcodec/image_decoder.h, is: parse the JPEG code stream, compute an in-bounds crop, fill nvimgcodecCodeStreamView_t{ .image_idx = 0, .region = { NVIMGCODEC_STRUCTURE_TYPE_REGION, sizeof(nvimgcodecRegion_t), nullptr, /*ndim=*/2, start[2], end[2] } }, call nvimgcodecCodeStreamGetSubCodeStream(), then decode with nvimgcodecDecodeParams_t.apply_exif_orientation set from the operator's adjust_orientation argument.

Environment details

tools/print_env.sh could not be run: the failures were observed on multi-node batch-scheduled CI clusters with no interactive shell on the affected nodes, and the environment used to prepare this report has no GPU. The closest captured environment evidence, taken from a controlled comparison of the passing and failing container environments:

Failing:  nvidia-nvimgcodec-cu13  0.9.0.1859   (with DALI 2.3.0)
Passing:  nvidia-nvimgcodec-cu13  0.8.0.1422   (with DALI 2.2.0)

CUDA major version:  13  (from the -cu13 wheel variant)
GPUs affected:       A100 80GB, H100 80GB, B200, GB200 NVL, GB300 NVL
Workload:            ImageNet-1k JPEG training data, DALI file reader +
                     fn.decoders.image_random_crop(device="mixed"),
                     batch size 128, 1 GPU
Failure timing:      first batch (during iterator prefetch)

The two versions were bundled together by a single container update, so we cannot offer a one-variable
nvImageCodec-only A/B; see Other/Misc.

Relevant log output

A100 / GB200 / GB300 — decode returns status 8:

Error in MIXED operator `nvidia.dali.fn.decoders.image_random_crop`,
which was used in the pipeline definition with the following traceback:

  File "...", line 46, in training_pipe
    images = fn.decoders.image_random_crop(

encountered the following error:

nvImageCodec failure: '#8'

RuntimeError: Critical error in pipeline:

The DALI frames leading in are the iterator's first-batch prefetch:
DALIClassificationIterator.__init__ -> DALIGenericIterator.__next__ ->
_get_outputs -> Pipeline.share_outputs -> Pipeline.ShareOutputs.

H100 / B200 — same pipeline, same wheel, crash instead:

Segmentation fault      (core dumped)  ./main.py ... --data-backend dali-gpu ...
exit_code='139'

Also present in every log, including successful 0.8.0 runs, and therefore
not related to this bug:

[ERROR] [nvtiff_cuda_decoder] Could not create nvtiff decoder:
nvTiff call failed with code 6: nvtiffStreamCreate(&nvtiff_stream_)

Other/Misc.

Regression boundary. Passing: nvImageCodec 0.8.0.1422 with DALI 2.2.0. Failing: nvImageCodec 0.9.0.1859 with DALI 2.3.0. A single container update bumped both, and we could not decouple them: DALI enforces a minimum runtime nvImageCodec version equal to the version it was compiled against, so DALI 2.3.0 cannot be run against 0.8.0.1422. nvImageCodec also publishes only squashed release-drop commits with no per-commit artifacts, so no finer bisection than the release boundary is possible from outside. Narrowing this further needs someone who can build the two components independently.

Why we think status 8 originates in the C API layer, not the extension. In src/image_generic_decoder.cpp, processBatchImpl discards an extension's returned status and converts the failure into NVIMGCODEC_PROCESSING_STATUS_FAIL, and ImageGenericDecoder::decode is noexcept and maps escaped exceptions the same way. So an extension-level decode failure should not surface to the caller as nvimgcodecStatus_t == 8; a returned 8 points at the C API entry points (NVIMGCODECAPI_CATCH maps any exception to INTERNAL_ERROR). Independently of this bug, silently dropping the extension's status loses useful diagnostics, and src/code_stream.cpp's static_get_image_info / static_get_codestream_info trampolines appear to return NVIMGCODEC_STATUS_SUCCESS unconditionally.

Test-coverage observation. test/extensions/roi_orientation_integration_test.cpp does cover this shape of request — InBoundsRawCoordRoiDecodesWithoutOrientation sends a region through nvimgcodecCodeStreamGetSubCodeStream with apply_exif_orientation = 0 across nvjpeg_cuda and nvjpeg_hw rows — so the gap is in the shape of the coverage rather than its absence:

• every ROI test constructs the decoder with ":fancy_upsampling=0", so no ROI test runs with default options, and fancy upsampling is precisely what canDecode gates ROI support on in both decoders;
• batch size 1 and a single backend only, so the batch_size > 1 branch of chooseDecodePath and the multi-sample legacy-batch ROI code are untested for ROI;
• no NVIMGCODEC_BACKEND_KIND_GPU_ONLY row;
• the nvjpeg_hw row is GTEST_SKIPped when the HW decoder or backend cannot be created, so on a CI GPU with at most one HW JPEG engine it passes green while testing nothing — meaning whichever of Legacy/BatchSingle the CI GPU does not select is never exercised for ROI. Given that we observe behavior splitting on exactly that axis, this looks like the gap most likely to have let the regression through.
• test/api/bitstream_offset_test.cpp is TIFF-only and performs no decode.

Cheapest checks to separate the candidates (we could not run these ourselves: they need either an instrumented DALI build or nvImageCodec-level hooks not reachable through DALI's Python API, and our reporting environment has no GPU):

1. Record num_hw_engines_ on an erroring GPU and on a segfaulting GPU. If it correlates with the Legacy/BatchSingle split at chooseDecodePath, candidate 1 is confirmed.
2. Re-run the failing case with ":fancy_upsampling=0" — the setting every ROI test uses. If it passes, the untested default-options ROI path is implicated.
3. Confirm which C API call returns 8, and print sizeof(nvimgcodecCodeStreamView_t) from the caller's translation unit, to rule the struct-layout change in or out.

Coordinate-space documentation gap. The public header documents nvimgcodecCodeStreamView_t::region only as /**< Region of interest. */. It does not state whether region is expressed in raw codestream coordinates or in EXIF-oriented display coordinates. src/imgproc/region_orientation.h's is_region_out_of_bounds_effective is self-consistent (raw dims when apply_exif_orientation is false, oriented dims when true), but callers have to infer the contract from the implementation. Worth documenting explicitly. Separately, the comment in src/imgproc/roi.h ("0 dimension is interpreted along x axis") reads as contradicting the start[0] = y convention used at the decoder call sites; we believe it describes a different internal type, but it is confusing.

Caller context. The sub-code-stream ROI usage on the DALI side was introduced by NVIDIA/DALI PR #6426; DALI 2.2.0 did not use this API. DALI declares its nvImageCodec dependency as nvidia-nvimgcodec-cu${CUDA_VERSION_MAJOR}[all] with no version constraint. DALI maintainers may want to confirm the region contract from their side.

Check for duplicates

• I have searched the open bugs/issues and have found no duplicates for this bug report. All 39 issues were reviewed; the newest is Tiled JPEG2000 Decoder Performance Regression #53 (2026-07-02, "Tiled JPEG2000 Decoder Performance Regression") and none concerns ROI, sub-code-streams, or EXIF orientation. The newest commit on the default branch is the v0.9.0 release drop (2026-07-14), so no post-0.9.0 fix is published.

---

This issue was drafted with assistance from the opus AI model.

[JanuszL]

This issue is related to unreleased 0.9.0.1859 version, while the public one 0.9.0.20 works as expected (I cannot repro it with it, while I can with the older).