Skip to content

[ExecuTorch][WebGPU] Op-test codegen framework (cases.py -> generated .pte+golden -> gtest driver)#20339

Closed
JulianCloudNTH wants to merge 3 commits into
gh/JulianCloudNTH/31/basefrom
gh/JulianCloudNTH/31/head
Closed

[ExecuTorch][WebGPU] Op-test codegen framework (cases.py -> generated .pte+golden -> gtest driver)#20339
JulianCloudNTH wants to merge 3 commits into
gh/JulianCloudNTH/31/basefrom
gh/JulianCloudNTH/31/head

Conversation

@JulianCloudNTH

@JulianCloudNTH JulianCloudNTH commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

A manifest-driven op-test framework for the WebGPU backend, mirroring Vulkan's op_tests/cases.py (declarative per-op suites) but with a torch-computed golden loaded in C++, since the native test binary has no ATen. An op fits when it is stateless and expressible as one module(inputs) -> golden forward; stateful KV-cache ops (sdpa, update_cache) stay hand-written. Lands first as the shared foundation for the following op test-diffs — adding an op's test becomes one cases.py entry.

Composition:

  • test_suite.py — schema (WebGPUTestSuite/Case/InputSpec) + a register_op_test decorator; per-case required/heavy/golden_fn, per-suite golden_dtype.
  • cases.py — the declarative suites; registers the landed add + rms_norm; later ops append one entry each.
  • generate_op_tests.py — per case: export via VulkanPartitioner to .pte, compute the fp64 torch golden (dual-oracle gate), serialize inputs+golden as fp32, emit manifest.json.
  • op_test_driver.cpp + driver_util.{h,cpp} — generic gtest driver: one test per manifest entry, runs forward on-device, abs/rel + shape + reconciliation checks.
  • CMakeLists/ci.shwebgpu_op_test + device-free webgpu_op_test_util_test, wired into the Dawn(Tint)+SwiftShader CI.
    @exported-using-ghexport

Differential Revision: D108816389

Differential Revision: D108816389

[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20339

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 6 Pending, 1 Unrelated Failure

As of commit b9f80dc with merge base eb7473b (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 17, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]
JulianCloudNTH added a commit that referenced this pull request Jun 17, 2026
… .pte+golden -> gtest driver)

Pull Request resolved: #20339

A manifest-driven op-test framework for the WebGPU backend, mirroring Vulkan's `op_tests/cases.py` (declarative per-op suites) but with a torch-computed golden loaded in C++, since the native test binary has no ATen. An op fits when it is stateless and expressible as one `module(inputs) -> golden` forward; stateful KV-cache ops (`sdpa`, `update_cache`) stay hand-written. Lands first as the shared foundation for the following op test-diffs — adding an op's test becomes one `cases.py` entry.

Composition:
- `test_suite.py` — schema (`WebGPUTestSuite`/`Case`/`InputSpec`) + a `register_op_test` decorator; per-case `required`/`heavy`/`golden_fn`, per-suite `golden_dtype`.
- `cases.py` — the declarative suites; registers the landed `add` + `rms_norm`; later ops append one entry each.
- `generate_op_tests.py` — per case: export via `VulkanPartitioner` to `.pte`, compute the fp64 torch golden (dual-oracle gate), serialize inputs+golden as fp32, emit `manifest.json`.
- `op_test_driver.cpp` + `driver_util.{h,cpp}` — generic gtest driver: one test per manifest entry, runs forward on-device, abs/rel + shape + reconciliation checks.
- `CMakeLists`/`ci.sh` — `webgpu_op_test` + device-free `webgpu_op_test_util_test`, wired into the Dawn(Tint)+SwiftShader CI.
ghstack-source-id: 394444416
@exported-using-ghexport

Differential Revision: [D108816389](https://our.internmc.facebook.com/intern/diff/D108816389/)
[ghstack-poisoned]
@meta-codesync

meta-codesync Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

This pull request has been merged in 6f4c21e.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants