Reset per-run state between variations of mlc test script

[anandhu-eng]

Problem

Every process_modified_files job in mlperf-automations#1086 whose script has both an r2-downloader and an rclone variation fails (example run):

* mlcr get,dataset,openorca,language-processing,preprocessed,_rclone,_mlc,_dry-run
  * mlcr download-and-extract,_r2-downloader,_rclone,_url.mlc-inference:mlcommons-inference-wg-public/open_orca
Error : Multiple variation tags selected for the variation group "download-tool": {'r2-downloader', 'rclone'}

The invoked tags are correct — _rclone only. The _r2-downloader comes from the previous variation in the same test.

Root cause

ScriptAutomation.run() falls back to the instance attributes whenever the caller doesn't pass the matching key:

https://github.com/mlcommons/mlcflow/blob/d6c103c/automation/script/module.py#L360-L369

self.env = i.get('env', self.env)
self.state = i.get('state', self.state)
self.const = i.get('const', self.const)
self.const_state = i.get('const_state', self.const_state)
self.add_deps_recursive = i.get('adr', self.add_deps_recursive)

Each CLI invocation gets a fresh ScriptAutomation from call_script_module_function, so a normal mlcr run is unaffected. But the test action loops over tests.run_inputs[*].variations_list calling self.run(ii) on one instance, and ii carries no adr. So:

1. _r2-downloader run merges {dae: {tags: '_r2-downloader'}} into self.add_deps_recursive.
2. _rclone run inherits it and merges _rclone on top.
3. The dae dep resolves as _r2-downloader,_rclone → two tags in the download-tool group.

self.recursion_spaces, self.env and self.run_state leak the same way — visible in the CI log as the second variation's output being indented one level deeper than the first.

This isn't specific to the two rclone-removal PRs; it hits any script whose variations_list has two entries from the same add_deps_recursive-carrying variation group. It surfaced now because #1086 touches 37 metas at once.

Fix

Snapshot the per-run state once in test() and restore it before each self.run(), so every variation starts from the state the automation object was constructed with. Scoped to the test loop — run() itself is untouched.

Test plan

Against mlcommons/mlperf-automations@60d0eb2 (main) with this mlcflow branch:

• mlc test script 5614c39cb1564d72 --test_input_index=1 --quiet (get-preprocessed-dataset-openorca, the failing job above) → exit 0. Deps now resolve as download-and-extract,_r2-downloader,_url.https://inference.mlcommons-storage.org/... and download-and-extract,_rclone,_url.mlc-inference:..., each with one download-tool tag.
• Same command on main without the fix reproduces the CI error verbatim, so the reproduction is faithful and the fix is what closes it.
• New regression test tests/test_script_test_state_isolation.py builds a throwaway two-script repo in a temp MLC_REPOS and asserts the second variation resolves its dep with _b alone. Fails on main, passes here.
• Full suite: python3 -m pytest tests/ -q → 101 passed.
• autopep8 -a --diff on both files reports nothing on the added lines (one pre-existing f-string elsewhere in module.py is untouched).

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[github-actions]

🤖 AI PR Review Summary

This PR introduces explicit snapshot and restore methods for the per-run state in ScriptAutomation to prevent state leakage between test variations. It addresses a subtle bug where instance attributes used as fallback in run calls caused state bleed across runs, leading to incorrect dependency selections. The design is sound and the added test verifies isolation. The main risk is ensuring all relevant state keys are included in PER_RUN_STATE_KEYS and that deep copies are sufficient to isolate mutable state. No major design issues detected.

[github-actions]

Consider adding a comment or docstring to PER_RUN_STATE_KEYS explaining why these specific keys are included and if this list needs to be updated when new per-run state attributes are added.

[github-actions]

Restoring the run state before each variation is critical; consider adding a comment here explaining that this prevents state bleed between variations for future maintainers.