Fix conda custom operator build and self-tests by JanuszL · Pull Request #6474 · NVIDIA/DALI

JanuszL · 2026-09-09T05:40:39Z

Fix conda custom-operator builds and self-tests.

Category:

Bug fix

Description:

Make DALI's Conda integration consistently select artifacts from the active
environment while retaining the existing wheel and local-build fallbacks:

Select $CONDA_PREFIX/include/dali only when the imported DALI package is
also under that prefix and libdali-devel supplies the headers.
Continue linking against the package-local prebuilt libraries, and install
public .hpp headers with the existing public .h headers.
Centralize C++ self-test binary discovery in find_test_bin. All eight
affected suites use it, so a missing binary cannot reuse a path found in a
previous loop iteration. The helper returns a canonical executable path for
GoogleTest death tests; it selects the imported package first for a DALI
package under the active Conda prefix, and otherwise retains local-build-first
lookup.

Additional information:

Affected modules and functionalities:

nvidia.dali.sysconfig include flags for custom-operator builds.
Installation of public DALI C++ headers for libdali-devel.
C++ self-test binary discovery in the TL0, TL1 Conda, CPU-only, Tegra, and
multi-GPU QA suites.

Key points relevant for the review:

A Conda prefix is used only when it matches the imported DALI package;
mismatched PYTHONPATH and Conda installations retain package-local headers.
Conda-installed DALI packages are preferred by the Conda self-test, while
non-Conda suites continue to prefer the local build produced by their job.
The helper has one search-order implementation and returns a nonzero status
when no executable is found.
libdali.so intentionally remains package-local because the bindings package
contains the prebuilt libraries.

Tests:

New tests added
- Python tests
  - dali/test/python/test_plugin_manager.py: matching, mismatched, and
    missing-libdali-devel Conda header-path selection.
- GTests
- Benchmark
- Other
  - Shell syntax checks for the helper and all eight affected QA scripts.
  - Focused helper check for local-build-first, Conda-package-first, and
    missing-binary behavior.
Existing tests apply
- TL1 Conda self-test discovery and the custom-operator notebook build run in
  CI.
N/A

Checklist

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: N/A

JIRA TASK: N/A

copy-pr-bot · 2026-09-09T05:40:43Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Conda installs DALI headers from libdali-devel in the active prefix, but get_include_dir() only returned the wheel-style package directory. Custom-operator builds therefore could not find the public DALI headers. Use the conda include directory when the active prefix contains DALI headers. The package-local include directory remains the fallback, preserving the behavior of wheel and non-conda installations. The conda self-test also searched PATH for test executables, although the conda package installs them beneath the nvidia.dali package. Search the local build output and installed package in the same order as TL0_self-test, report a missing executable clearly, and retain an absolute path for GoogleTest death tests. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

greptile-apps · 2026-09-09T05:48:30Z

Confidence Score: 5/5

The PR appears safe to merge; no outstanding correctness or rule-compliance finding remains.

Summary

Selects Conda development headers only when they correspond to the imported DALI package.
Installs public .hpp headers alongside public .h headers.
Preserves local-build-first selection for TL0 suites while allowing the Conda suite to prefer its active package.
Improves missing-binary diagnostics with the requested binary and searched paths.

_{Reviews (8) · Last reviewed commit: "Clarify self-test binary lookup failures"}

dali-automaton · 2026-09-09T05:48:40Z

CI MESSAGE: [66908778]: BUILD STARTED

Only use headers from CONDA_PREFIX when the imported nvidia.dali package is located under that same prefix. This prevents custom operators from compiling against conda headers while linking to a separately installed DALI wheel. Add focused sysconfig tests for both matching and mismatched installations. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

JanuszL · 2026-09-09T05:53:02Z

@greptile review

Compute the conda include path only where it is needed. This keeps the matching-installation guard while avoiding CodeQL's unused-local diagnostic. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

JanuszL · 2026-09-09T05:59:28Z

@greptile review

dali-automaton · 2026-09-09T06:42:32Z

CI MESSAGE: [66916279]: BUILD STARTED

dali-automaton · 2026-09-09T08:24:20Z

CI MESSAGE: [66916279]: BUILD FAILED

dali-automaton · 2026-09-09T11:46:48Z

CI MESSAGE: [66959790]: BUILD STARTED

dali-automaton · 2026-09-09T16:04:21Z

CI MESSAGE: [66959790]: BUILD PASSED

Prefer self-test binaries from the DALI package imported by the active Conda environment, before considering a local build-tree fallback. Canonicalize the chosen executable for death tests and preserve the suite's existing executor coverage. Document why Conda headers come from libdali-devel while prebuilt libraries remain with the Python package. Add coverage for the fallback used when the imported DALI package is in Conda but libdali-devel is not installed. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

dali-automaton · 2026-09-09T21:59:44Z

CI MESSAGE: [67055128]: BUILD STARTED

Install public .hpp headers alongside .h headers so that the installed development package contains the complete include closure required by float16.h and custom DALI operators. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

dali-automaton · 2026-09-09T23:10:23Z

CI MESSAGE: [67065835]: BUILD STARTED

dali-automaton · 2026-09-10T02:32:23Z

CI MESSAGE: [67065835]: BUILD PASSED

jantonguirao

All five review comments from the previous pass were addressed — binary discovery now checks the actual conda install layout, the death-test path comment matches the readlink -f behavior, the incidental DALI_USE_EXEC2=0 change was reverted, get_lib_dir's conda-vs-wheel split is now documented, and the missing test branch (package inside $CONDA_PREFIX without libdali-devel) is covered. No new issues found on re-review. Leaving one non-blocking design note below about the discovery-loop duplication across QA scripts.

Use a shared helper for every self-test binary-discovery loop. It canonicalizes selected executables, reports missing binaries without reusing a previous loop iteration, and preserves the active Conda package priority. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Make Conda package priority an explicit helper option used only by the TL1 Conda suite. TL0 suites continue testing their current job local build even when a Conda environment is active. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Report the missing binary and every searched location so failed QA self-test discovery is immediately diagnosable. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

dali-automaton · 2026-09-10T10:41:34Z

CI MESSAGE: [67158752]: BUILD STARTED

dali-automaton · 2026-09-10T13:39:31Z

CI MESSAGE: [67158752]: BUILD PASSED

JanuszL force-pushed the fix-conda-self-test-binary-discovery branch from 4cda17c to e16ab34 Compare September 9, 2026 05:43

JanuszL force-pushed the fix-conda-self-test-binary-discovery branch from e16ab34 to bc678c1 Compare September 9, 2026 05:45

greptile-apps Bot reviewed Sep 9, 2026

View reviewed changes

Comment thread dali/python/nvidia/dali/sysconfig.py Outdated

github-advanced-security Bot found potential problems Sep 9, 2026

View reviewed changes

Comment thread dali/python/nvidia/dali/sysconfig.py Fixed

Simplify conda include path selection

349a087

Compute the conda include path only where it is needed. This keeps the matching-installation guard while avoiding CodeQL's unused-local diagnostic. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

dali-automaton assigned rostan-t and jantonguirao Sep 9, 2026

jantonguirao reviewed Sep 9, 2026

View reviewed changes

Comment thread qa/TL1_self-test_conda/test.sh Outdated

Comment thread qa/TL1_self-test_conda/test.sh

Comment thread qa/TL1_self-test_conda/test.sh Outdated

Comment thread dali/python/nvidia/dali/sysconfig.py

Comment thread dali/test/python/test_plugin_manager.py

Install public C++ headers

42d06d6

Install public .hpp headers alongside .h headers so that the installed development package contains the complete include closure required by float16.h and custom DALI operators. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

jantonguirao approved these changes Sep 10, 2026

View reviewed changes

Comment thread qa/TL1_self-test_conda/test.sh Outdated

JanuszL force-pushed the fix-conda-self-test-binary-discovery branch from a06767f to 92a59f2 Compare September 10, 2026 10:13

greptile-apps Bot reviewed Sep 10, 2026

View reviewed changes

Comment thread qa/setup_test_common.sh Outdated

Preserve TL0 self-test binary priority

5909417

Make Conda package priority an explicit helper option used only by the TL1 Conda suite. TL0 suites continue testing their current job local build even when a Conda environment is active. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

greptile-apps Bot reviewed Sep 10, 2026

View reviewed changes

Comment thread qa/setup_test_common.sh

Clarify self-test binary lookup failures

e83ea22

Report the missing binary and every searched location so failed QA self-test discovery is immediately diagnosable. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

rostan-t approved these changes Sep 10, 2026

View reviewed changes

JanuszL merged commit 3e40f6d into NVIDIA:main Sep 10, 2026
9 checks passed

JanuszL deleted the fix-conda-self-test-binary-discovery branch September 10, 2026 15:38

Uh oh!

Conversation

JanuszL commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Uh oh!

copy-pr-bot Bot commented Sep 9, 2026

Uh oh!

greptile-apps Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

dali-automaton commented Sep 9, 2026

Uh oh!

JanuszL commented Sep 9, 2026

Uh oh!

Uh oh!

JanuszL commented Sep 9, 2026

Uh oh!

dali-automaton commented Sep 9, 2026

Uh oh!

dali-automaton commented Sep 9, 2026

Uh oh!

dali-automaton commented Sep 9, 2026

Uh oh!

dali-automaton commented Sep 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dali-automaton commented Sep 9, 2026

Uh oh!

dali-automaton commented Sep 9, 2026

Uh oh!

dali-automaton commented Sep 10, 2026

Uh oh!

jantonguirao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dali-automaton commented Sep 10, 2026

Uh oh!

dali-automaton commented Sep 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JanuszL commented Sep 9, 2026 •

edited

Loading

greptile-apps Bot commented Sep 9, 2026 •

edited

Loading