Add data-loading bottleneck diagnostic skill

[rostan-t]

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

Create a skill allowing agent to detect and localize data loading bottlenecks in PyTorch training.

The workflow roughly contains two important phases:

• Detection of whether or not there's a data loading bottleneck using DALI's LoaderEvaluator
• Bottleneck localization with profiling

The complete workflow is more complex and looks as follows:

1. Identify the workload, environment and if it's at all possible to run.
2. Run (bounded) training, measure loader wait. If LoaderEvaluator is or can be made importable , continue to 3. Else, go to 4.
3. Replay the training with cached batches using LoaderEvaluator's replay mode. Establish if there's a measured bottleneck. If we this steps provdes that there's no bottleneck, go to 5. Otherwise, continue to 4. to localize.
4. Run profiling to localize the bottleneck if we come from 3., or to detect it if coming from 2.
5. Report the results.

This was tested on multiple models, with different environment requirements, including simple ResNet50, OpenCLIP, LeWorldModel, and Hugging Face timm.

In terms of the model used by the agent, I found that GPT 5.6 Terra or equivalent is a sound minimum requirement to run this reliably, although it can sometimes fail to accurately follow all instructions, like the shape of the report.

I unfortunately couldn't commit evaluations as the skill CI doesn't support GPU environments.

Additional information:

Affected modules and functionalities:

Skill created.

Key points relevant for the review:

• Is the skill's workflow sound?
• Can the skill miss an existing bottlenecks or detect one where there isn't?
• Is the skill too defensive on things like requiring equivalent work for replay?

Note that if you run locally, depending on the workload, it may take time to complete.

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Skill CI doesn't support GPU workers yet

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other (skill)
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-4796

[greptile-apps]

Greptile Summary

The PR adds an agent skill for diagnosing PyTorch data-loading bottlenecks through preflight checks, bounded Real/Replay comparisons, and Nsight Systems profiling.

• Adds workflow instructions, profiling and DALI replay references, and a structured report template.
• Adds environment/evaluation definitions for input-bound, compute-bound, and out-of-scope workloads.
• Adds scripts for collecting environment readiness and summarizing Nsight trace data.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
skills/data-loading-bottleneck/SKILL.md	Defines the end-to-end diagnostic routing, equivalence requirements, verdict thresholds, and reporting contract.
skills/data-loading-bottleneck/references/profiling.md	Documents production-path NVTX instrumentation, Nsight capture validation, and evidence requirements for localization.
skills/data-loading-bottleneck/references/pytorch-dali.md	Documents bounded LoaderEvaluator construction and Real/Replay work-equivalence checks.
skills/data-loading-bottleneck/scripts/collect_preflight.py	Collects workload, environment, GPU, dependency, filesystem, and Git readiness into a preflight artifact.
skills/data-loading-bottleneck/scripts/summarize_nsys.py	Exports and summarizes the selected NVTX window, CUDA activity, worker coverage, and loader-wait overlap.
skills/data-loading-bottleneck/evals/evals.json	Defines representative input-bound, compute-bound, and inference scenarios for evaluating skill behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Preflight workload and environment] --> B{Workload runnable?}
    B -- No --> I[Report INCONCLUSIVE]
    B -- Yes --> C[Run bounded Real workload]
    C --> D{LoaderEvaluator available?}
    D -- Yes --> E[Run equivalent Replay]
    E --> F{Replay speedup}
    F -- <= 1.10x --> N[Report NOT DETECTED]
    F -- > 1.10x --> P[Profile production path]
    D -- No --> P
    P --> G{Validated input-path delay?}
    G -- Yes --> H[Report DETECTED or POTENTIAL and localize]
    G -- No --> I

Loading

_{Reviews (3): Last reviewed commit: "Add eval suite for data-loading bottlene..." | Re-trigger Greptile}

[JanuszL]

/nvskills-ci

[JanuszL]

/nvskills-ci

[mdabek-nvidia]

Is the intent to update version number manually? Can it re-use DALI versioning ?

[rostan-t]

Since skills are distributed independently from DALI (consumed by NVIDIA/skills), I wouldn't be sure what version to use here. Should it be 2.3.0 or the current version in VERSION 2.4.0dev? Do we open a PR to update the skill versions each time we update DALI?

[mdabek-nvidia]

You are right, that since it is distributed independently it would be difficult to share the versioning. I think that we need to keep in mind, that the skill itself may need to be updated after changes in DALI. I would feel safer knowing what is minimal version of DALI that would be supported by the skill.

[mdabek-nvidia]

The skill assumes that the platform that we run the preflight is configured so that it can run training.
Would it be possible to extend it to create virtual environment and execute pip install -f requiremnets.txt if such file exist?

[rostan-t]

I think it's generally safe to assume that somebody interested in evaluating if training performance is bottlenecked by data loading has a working environment. We want to use this environment with the exact pinned library versions they are using. Those might not be the same as a potential requirements.txt file or it might not tell the full picture because it's not uncommon for projects to have other files like requirements.dev.txt.

Additionally, supporting requirements.txt only would be incomplete, pyproject.toml is also often used (with e.g. uv or poetry), some projects have a conda configuration instead, others rely on Docker containers, etc.

[mdabek-nvidia]

I think it's generally safe to assume that somebody interested in evaluating if training performance is bottlenecked by data loading has a working environment.

This is bold assumption. There are cases where production environment is isolated from development environment with running agent. I think that adding this assumption (or asking a user to point to an active environment) to the skill would make it more user friendly.