Skip to content

fix(ci): enforce post_training isolation in test coordinator and fix missing test markers across test files - #4721

Merged
copybara-service[bot] merged 1 commit into
mainfrom
maxtext-test-marker-isolation
Aug 6, 2026
Merged

fix(ci): enforce post_training isolation in test coordinator and fix missing test markers across test files#4721
copybara-service[bot] merged 1 commit into
mainfrom
maxtext-test-marker-isolation

Conversation

@RexBearIU

@RexBearIU RexBearIU commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR fixes leaky test marker expressions in the CI test coordinator and adds missing Pytest markers across 6 test files in MaxText, ensuring pre-training and post-training test suites are strictly isolated and do not accidentally cross-trigger.

Why this change is being made & problem being solved

In .github/workflows/run_tests_coordinator.yml, tpu-post-training-integration and tpu7x-post-training-integration previously checked 'not cpu_only and not gpu_only and integration_test'. Because and post_training was omitted from the filter expression, pre-training integration tests without a post_training mark were leaking into the post-training integration test suite. Furthermore, several test files under tests/post_training/ and tests/integration/ lacked explicit directory/file-level marker declarations (pytestmark = [pytest.mark.post_training], pytestmark = pytest.mark.integration_test), leading to incomplete test suite tagging.

Specific implementation details

  1. CI Coordinator Filter Isolation: Added and post_training to tpu-post-training-integration and tpu7x-post-training-integration marker expressions in .github/workflows/run_tests_coordinator.yml.
  2. Post-Training Unit Test Tagging: Added pytestmark = pytest.mark.post_training to 3 post-training unit tests:
    • tests/post_training/unit/learn_to_init_test.py
    • tests/post_training/unit/vllm_decode_test.py
    • tests/post_training/unit/vllm_rollout_unroll_test.py
  3. Integration Test Tagging: Added pytest.mark.integration_test to 3 integration test files:
    • tests/post_training/integration/grpo_correctness.py
    • tests/integration/check_vma_test.py
    • tests/integration/aot_identical_test.py
  4. Engine Isolation Safeguard: Confirmed Native LoRA (use_sft=True in lora_e2e_nnx_test.py) remains marked not post_training as it operates on the pre-training engine.
  5. Strict Marker Compliance: Preserved default unit test inheritance (not integration_test and not post_training) for tests/unit/ without introducing unregistered marker tags that would violate --strict-markers in pytest.ini.

Shortcomings & future improvements

In future work, we can consider splitting tpu-tests in ci_pipeline.yml into separate matrix jobs for pre-training and post-training suites to enable direct domain-level job gating.

Tests

  • Pre-commit Validation: Passed cleanly on all modified files:
    pre-commit run --files .github/workflows/run_tests_coordinator.yml \
      tests/post_training/unit/learn_to_init_test.py \
      tests/post_training/unit/vllm_decode_test.py \
      tests/post_training/unit/vllm_rollout_unroll_test.py \
      tests/post_training/integration/grpo_correctness.py \
      tests/integration/check_vma_test.py \
      tests/integration/aot_identical_test.py
  • CI Test Suite Runs: All required CI workflows completed successfully on GitHub Actions (36/37 checks passing; All Required Tests Passed = Success):
    • cpu-unit tests: PASSED
    • gpu-unit tests: PASSED
    • tpu-unit tests: PASSED
    • cpu-post-training-unit tests: PASSED
    • tpu-post-training-unit tests: PASSED
    • gpu-integration tests: PASSED
    • tpu-integration tests: PASSED
    • maxtext_tpu_pathways_unit_tests: PASSED
    • maxtext_tpu_pathways_integration_tests: PASSED
  • Strict Marker Audit: Verified --strict-markers compliance against pytest.ini.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@RexBearIU RexBearIU changed the title fix(ci): fix missing test markers across 6 test files and enforce post_training isolation in test coordinator fix(ci): enforce post_training isolation in test coordinator and fix missing test markers across 6 test files Aug 4, 2026
@RexBearIU RexBearIU changed the title fix(ci): enforce post_training isolation in test coordinator and fix missing test markers across 6 test files fix(ci): enforce post_training isolation in test coordinator and fix missing test markers across test files Aug 4, 2026
@RexBearIU
RexBearIU force-pushed the maxtext-test-marker-isolation branch from 4c34f48 to a606bf5 Compare August 6, 2026 08:14
@RexBearIU
RexBearIU requested a review from shuningjin as a code owner August 6, 2026 08:14
@RexBearIU
RexBearIU force-pushed the maxtext-test-marker-isolation branch from a606bf5 to b6b9589 Compare August 6, 2026 08:59
@copybara-service
copybara-service Bot merged commit 5825af5 into main Aug 6, 2026
49 checks passed
@copybara-service
copybara-service Bot deleted the maxtext-test-marker-isolation branch August 6, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants