Skip to content

Do not hold the global BleedingEdgeToggle across yield in data providers - fixes racy/flaky tests#5911

Merged
staabm merged 18 commits into
phpstan:2.2.xfrom
phpstan-bot:create-pull-request/patch-m6t0ssr
Jun 22, 2026
Merged

Do not hold the global BleedingEdgeToggle across yield in data providers - fixes racy/flaky tests#5911
staabm merged 18 commits into
phpstan:2.2.xfrom
phpstan-bot:create-pull-request/patch-m6t0ssr

Conversation

@phpstan-bot

Copy link
Copy Markdown
Collaborator

Summary

PHPStan\Type\IntersectionTypeTest::testIsAcceptedBy failed intermittently in CI
(most often on Windows / PHP 7.4 / 8.0):

non-empty-array&hasOffsetValue(0, int) -> isAcceptedBy(array{int, int})
-'Maybe'
+'No'

The failure is a test-isolation problem, not a type-system bug. The data providers
of ConstantArrayTypeTest mutate the global static BleedingEdgeToggle and keep
it mutated across yield boundaries. Because data providers are evaluated lazily and
can be abandoned mid-iteration (filtering, random ordering, early stop), the global was
left stuck on true and leaked into unrelated tests.

When the toggle leaks as true, every ConstantArrayType constructed afterwards — including
the array{int, int} built by IntersectionTypeTest::dataIsAcceptedBy — becomes a sealed
array shape. ConstantArrayType::accepts() then short-circuits a non-constant-array operand
with No ("Sealed array shape can only accept a constant array. Extra keys are not allowed.")
instead of the correct Maybe. Hence the intermittent MaybeNo flip.

The fix makes the providers build their toggle-dependent data sets eagerly and restore the
global before yielding, so the toggle is never observable as mutated outside the provider call.

Changes

  • tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php
    • Added dataWithBleedingEdge(bool $bleedingEdge, callable $cases): array — sets the toggle,
      invokes the callback to build the data sets while the toggle is set, and restores the
      previous value in a finally before returning.
    • Rewrote dataAccepts() to produce its two bleeding-edge-dependent groups (unknown-sealedness
      and sealed) through dataWithBleedingEdge(...) + yield from, instead of toggling the global
      inline between yields.
    • Rewrote dataGetArraySize() the same way (its foreach ([false, true] ...) loop previously
      held the toggle across every yield, and even left it on true for the trailing group).
    • Added testDataProviderDoesNotLeakBleedingEdgeToggle() (data-provided over both providers)
      that partially consumes each provider for every prefix length and asserts the toggle is
      restored — a deterministic reproduction of the flaky behavior.

Root cause

Pattern: a lazily-evaluated generator data provider mutates a global static and holds it
mutated across a yield.
When such a generator is abandoned before completion (or its
post-last-yield cleanup never runs), the global stays mutated for the rest of the process,
and every later object construction that reads that global is silently affected.

Concretely:

  • ConstantArrayType::__construct() reads BleedingEdgeToggle::isBleedingEdge() to decide the
    default sealedness (unsealed = [never, never] when bleeding edge is on).
  • ConstantArrayTypeTest::dataAccepts() and dataGetArraySize() flipped the toggle to true
    and yielded objects while it was held, restoring it only after the final yield.
  • A consumer that abandoned the iterator (empirically reproducible: stopping after 6+ data sets)
    left the global on true, so IntersectionTypeTest::dataIsAcceptedBy built sealed arrays and
    testIsAcceptedBy data set DOMNode child and sibling accessor properties should be nullable #7 returned No.

The fix removes the only two places where this pattern occurred. The object construction still
happens under the requested toggle value (the callback runs inside the set/restore window), but
the global is restored synchronously before any value is yielded, so no consumer can ever observe
the mutated state.

Test

  • testDataProviderDoesNotLeakBleedingEdgeToggle() consumes dataAccepts() and
    dataGetArraySize() for every prefix length 1..60 and asserts BleedingEdgeToggle::isBleedingEdge()
    is false afterwards. Verified it fails before the fix (toggle leaks true after consuming
    6 data sets) and passes after.
  • IntersectionTypeTest, ConstantArrayTypeTest, and TypeCombinatorTest pass under repeated
    --order-by=random runs; the full suite (make tests) and make phpstan are green.

Analogous cases probed

  • Searched the whole test suite for global-state mutations inside generator data providers
    (BleedingEdgeToggle, ini_set, putenv, setlocale, static setters held across yield):
    the only leaking generators were these two providers.
  • TypeCombinatorTest::testUnion/testIntersect/... and ConstantArrayTypeTest::testIsSuperTypeOf/
    testSealedness/testEqualsTreatsLegacyNullAndSealedMarkerAsEqual toggle the global inside test
    methods with try/finally, and run after providers are evaluated — they cannot contaminate
    provider-time construction and were left unchanged.
  • The putenv() calls in the error-formatter tests live in setUp/tearDown/test methods, not
    providers, and are restored in tearDown.

Fixes phpstan/phpstan#14855

@staabm staabm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are 2 more failures. are these related and need the same fix?

There were 2 failures:

1) PHPStan\Type\Generic\GenericObjectTypeTest::testIsSuperTypeOf with data set #3 (PHPStan\Type\Generic\GenericObjectType Object (...), PHPStan\Type\Generic\GenericObjectType Object (...), PHPStan\TrinaryLogic Object (...))
ReflectionClass<stdClass> -> isSuperTypeOf(ReflectionClass<object>)
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Maybe'
+'No'

D:\a\phpstan-src\phpstan-src\tests\PHPStan\Type\Generic\GenericObjectTypeTest.php:275
D:\a\phpstan-src\phpstan-src\tests\vendor\phpunit\phpunit\src\TextUI\Command.php:146

2) PHPStan\Type\Generic\GenericObjectTypeTest::testIsSuperTypeOf with data set #2 (PHPStan\Type\Generic\GenericObjectType Object (...), PHPStan\Type\Generic\GenericObjectType Object (...), PHPStan\TrinaryLogic Object (...))
ReflectionClass<object> -> isSuperTypeOf(ReflectionClass<stdClass>)
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'
+'No'

D:\a\phpstan-src\phpstan-src\tests\PHPStan\Type\Generic\GenericObjectTypeTest.php:275
D:\a\phpstan-src\phpstan-src\tests\vendor\phpunit\phpunit\src\TextUI\Command.php:146

@phpstan-bot

Copy link
Copy Markdown
Collaborator Author

I've completed a thorough investigation. Here's my evidence-based answer to the reviewer's question.

Investigation summary

Short answer: No — those two GenericObjectTypeTest failures are not caused by the BleedingEdgeToggle leak, and the same fix does not address them. They are a separate test-isolation problem.

Why they look similar but aren't

Both failures show a result collapsing to 'No', which is what made the relation plausible. But the mechanisms are unrelated:

IntersectionTypeTest (this PR) GenericObjectTypeTest (reviewer's 2 failures)
Flip Maybe → No Yes → No (#2) and Maybe → No (#3)
Real cause leaked toggle makes ConstantArrayType sealed ReflectionClass<T> treated as invariant instead of covariant
Controlled by BleedingEdgeToggle PhpVersion::supportsLazyObjects() → which ReflectionClass stub is loaded

The two failing data sets are GenericObjectTypeTest.php:132-148 — both gated on PHP_VERSION_ID >= 80400 ? No : Yes/Maybe. The expectation differs by PHP version because the variance differs:

  • stubs/ReflectionClass.stub@template-covariant T (PHP < 8.4)
  • stubs/ReflectionClassWithLazyObjects.stub@template T invariant (PHP 8.4+)

Selected per-container by ReflectionClassStubFilesExtension based on supportsLazyObjects(). The reviewer's CI (7.4/8.0) saw the invariant (No) result — i.e. the process behaved as if the PHP 8.4 stub were active.

Proof the toggle is not involved

BleedingEdgeToggle::isBleedingEdge() is consulted only in ConstantArrayType / ConstantArrayTypeBuilder — never in generics, variance, or ReflectionClass:

$ grep -rln isBleedingEdge src/
src/DependencyInjection/BleedingEdgeToggle.php
src/Type/Constant/ConstantArrayTypeBuilder.php
src/Type/Constant/ConstantArrayType.php

I reproduced the exact scenario under a simulated PHP 8.0 container and toggled the global both ways:

toggle=false  #2(obj->std)=Yes  #3(std->obj)=Maybe
toggle=true   #2(obj->std)=Yes  #3(std->obj)=Maybe   ← toggle has zero effect

And I confirmed per-container isolation holds — building a PHP 8.4 container first does not leak invariant variance into a later PHP 7.4 container in the same process:

[php84-first]      variance=invariant  #2=No   #3=No
[php74-after-84]   variance=covariant  #2=Yes  #3=Maybe

So this PR's fix is correct and complete for the flakiness it targets; it neither causes nor cures the GenericObjectTypeTest failures.

What the GenericObjectTypeTest failures actually are

A separate isolation issue: on those runs the process resolved ReflectionClass with the PHP 8.4 lazy-objects (invariant) stub while running on 7.4/8.0. That points at supportsLazyObjects() / stub selection (most likely BetterReflection intermittently not applying the covariant stub and falling back to native reflection, which lacks @template-covariant — consistent with "mostly Windows, sometimes Linux"). I could not reproduce it on this PHP 8.4 machine, where the invariant result is the correct expectation, so I deliberately did not push a speculative change — per the repo's own rule, a fix must first be made to fail for the right reason, and weakening the assertion to hide it would mask a real bug.

Recommendation

Keep this PR scoped to the BleedingEdgeToggle leak (verified). The GenericObjectTypeTest variance flakiness should be tracked as its own issue against ReflectionClass stub selection under PHP 7.4/8.0.

I made no code changes (the working tree is clean) because none is warranted here and I can't verify one against a reproduction — so there's nothing to commit. If you'd prefer, I can open a separate investigation into the supportsLazyObjects()/stub-selection path instead.

@staabm staabm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of the current approach, could we extract a new PHPStanBleedingEdgeToggleTestCase class, which implements all necessary set/restore machinery for BleedingEdgeToggle and can be extended by all real use-cases which depend on BleedingEdgeToggle?

@phpstan-bot

Copy link
Copy Markdown
Collaborator Author

Done. I addressed the reviewer's request by extracting the set/restore machinery into a reusable base class.

What changed

New src/Testing/PHPStanBleedingEdgeToggleTestCase.php — a base test case (extending PHPStanTestCase) that centralizes all BleedingEdgeToggle set/restore logic:

  • withBleedingEdge(bool $bleedingEdge, callable $callback) — sets the toggle, runs the callback, and restores the previous value in a finally. Works both for test methods and for data providers that build their toggle-dependent objects eagerly inside the callback (so nothing is held across a yield).
  • tearDown() — resets the global default as a safety net, even if a test method or a partially consumed data provider left the toggle mutated.

Adopted across all real consumers of BleedingEdgeToggle:

  • ConstantArrayTypeTest — extends the base class; its private dataWithBleedingEdge() helper is gone, replaced by self::withBleedingEdge(...). The deterministic leak regression test is kept.
  • ConstantArrayTypeBuilderTest — extends the base class; inline try/finally replaced with withBleedingEdge().
  • TypeCombinatorTest — extends the base class; the four identical backup/set/restore blocks in testUnion/testUnionInversed/testIntersect/testIntersectInversed replaced with withBleedingEdge().

The two ConstantArrayTypeTest methods that legitimately flip the toggle multiple times mid-test (testSealedness, testEqualsTreatsLegacyNullAndSealedMarkerAsEqual) keep their explicit try/finally and now also benefit from the base-class tearDown() safety net.

Verification

  • make phpstan — no errors (added the required #[Override] on tearDown).
  • make tests — all green (17436 tests).
  • The leak regression test still passes; affected test files pass under the new base class.

Committed as two logical commits (introduce base class + migrate primary file; adopt in remaining files) and pushed.

Comment thread src/Testing/PHPStanBleedingEdgeToggleTestCase.php Outdated
Comment thread src/Testing/PHPStanBleedingEdgeToggleTestCase.php Outdated
Comment thread src/Testing/PHPStanBleedingEdgeToggleTestCase.php Outdated
@staabm

staabm commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

note to reviewer: there are still other intermittent errors, but these have a different root. will be tackled in a separate PR.

@staabm staabm requested a review from VincentLanglet June 21, 2026 17:52
@staabm staabm force-pushed the create-pull-request/patch-m6t0ssr branch from 966d15e to 3d3c222 Compare June 21, 2026 18:30
@staabm staabm removed the request for review from VincentLanglet June 21, 2026 18:34

@staabm staabm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PHP8 tests are still failling

1) PHPStan\Type\Constant\ConstantArrayTypeTest::testIsSuperTypeOf with data set #26 (PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\TrinaryLogic Object (...))
array{a: string} -> isSuperTypeOf(array{a: string, b: int})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'
+'No'

/home/runner/work/phpstan-src/phpstan-src/tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php:1011
/home/runner/work/phpstan-src/phpstan-src/tests/vendor/phpunit/phpunit/src/TextUI/Command.php:146

2) PHPStan\Type\Constant\ConstantArrayTypeTest::testIsSuperTypeOf with data set #10 (PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\TrinaryLogic Object (...))
array{foo: int} -> isSuperTypeOf(array{foo: int, bar: int})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'
+'No'

/home/runner/work/phpstan-src/phpstan-src/tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php:1011
/home/runner/work/phpstan-src/phpstan-src/tests/vendor/phpunit/phpunit/src/TextUI/Command.php:146

3) PHPStan\Type\Constant\ConstantArrayTypeTest::testAccepts with data set #12 (PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\TrinaryLogic Object (...))
array{foo: string} -> accepts(array{foo: string, bar: string})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'
+'No'

/home/runner/work/phpstan-src/phpstan-src/tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php:632
/home/runner/work/phpstan-src/phpstan-src/tests/vendor/phpunit/phpunit/src/TextUI/Command.php:146

4) PHPStan\Type\Constant\ConstantArrayTypeTest::testAccepts with data set #25 (PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\TrinaryLogic Object (...))
array{name?: string, color?: string} -> accepts(array{sound: 'test'})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'

@phpstan-bot

Copy link
Copy Markdown
Collaborator Author

Done. Here's a summary of what I found and fixed.

Root cause

The reviewer's 4 PHP8 failures (testIsSuperTypeOf #10/#26, testAccepts #12/#25) are the same test-isolation bug as the original issue, but the previously-merged provider-yield fix didn't cover them.

ConstantArrayType::__construct() reads the global BleedingEdgeToggle to decide default sealedness. The leak source is ContainerFactory.php:204: every test that builds a bleeding-edge container calls setBleedingEdge(true) and never restores it. Several such tests live in tests/PHPStan/Type/ (e.g. TypeGetFiniteTypesTest, OversizedArrayBuilderTest, TypeToPhpDocNodeTest). In a paratest worker that runs one of those before ConstantArrayTypeTest, the toggle is left true when its data providers evaluate.

The failing data sets are the object-based sets in dataAccepts() and dataIsSuperTypeOf() — they were constructed under the ambient toggle and assume unsealed/legacy behavior. Under the leak they become sealed, flipping Yes/MaybeNo.

I reproduced this exactly: with the toggle forced true at provider-eval time, the old code produced 9 failures (including the reviewer's #10/#26); with the fix, 0.

Fix

Wrapped the object-block construction of both providers in the reviewer-blessed BleedingEdgeToggle::withBleedingEdge(false, static fn (): array => [...]) + yield from, so the shapes are built with an explicit, deterministic toggle value regardless of any leak. The string-based sets (resolved later under withBleedingEdge(true)) and the existing sealed groups are unchanged. I also merged the two now-adjacent withBleedingEdge(false, ...) groups in dataAccepts().

Verification

  • ConstantArrayTypeTest + IntersectionTypeTest + TypeCombinatorTest + ConstantArrayTypeBuilderTest: 1477 pass
  • make tests: 17436 pass (96 skipped, as usual)
  • make phpstan: no errors
  • Proved the fix immune to a forced external leak; proved old code failed under the same leak.

Committed as 0a9fcd539 and pushed.

One note for the reviewer (not changed here)

Because the real leak source is ContainerFactory (not just provider-yields), IntersectionTypeTest::dataIsAcceptedBy — the literal test from #14855 — and other Type/*Test.php providers that build ConstantArrayType are technically still vulnerable to the same external leak. I kept this PR scoped to the providers you flagged, consistent with your "other intermittent errors → separate PR" note. If you'd prefer, the same withBleedingEdge(false, ...) wrap can be applied to those, or the toggle can be reset centrally — let me know which direction you want.

@VincentLanglet

Copy link
Copy Markdown
Contributor

note to reviewer: there are still other intermittent errors, but these have a different root. will be tackled in a separate PR.

I find hacky the need to add use withBleedingEdge(false) for the constantArrayTypeTest.
Shouldn't it be the default value ? Why isn't it restored correctly by previous tests ?

@staabm

staabm commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Why isn't it restored correctly by previous tests ?

my understanding is, that with the FNSR a test which is run from a dataprovider with yield, could be interruppted mid-flight and another one could start before the previous one will cleanup its global state.

code which runs yield will be paused and resumed. and since with FNSR we also use Fibers in phpstan-src we yield control between phpstan-src and the testsuite.

btw: as soon as we release PHPStan 3.0, the bleeding edge toggle in the ConstantArrayType impl will be gone and the tests can be cleaned up again

@staabm staabm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could the root problem also be fixed by avoiding yield in the data-provider and return one big array instead?
(so we would not need to adjust existing bleeding-edge toggle set/reset code like done in this PR)

@staabm

staabm commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

I find hacky the need to add use withBleedingEdge(false) for the constantArrayTypeTest.
Shouldn't it be the default value ?

I think its just about beeing explicit.

after the long block of yield from BleedingEdgeToggle::withBleedingEdge(false, static fn (): array => [ there is also another one with yield from BleedingEdgeToggle::withBleedingEdge(true, static fn (): array => [

I like the new code more than the previous one.

Comment on lines +41 to +43
if ($result instanceof Generator) {
throw new ShouldNotHappenException('callback is not allowed to yield, to prevent leaking the toggle into unrelated tests.');
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note to reviewers: this line is the one which prevents regressions. if we consistently stick with this API, new tests requiring bleeding edge cannot accidentally use generators which can introduce race-conditions for this problem

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The current review was a "note to reviewers" from staabm highlighting that the if ($result instanceof Generator) line in withBleedingEdge is the safety net that prevents the regression class this whole PR addresses — a data provider accidentally holding the toggle across a yield.

Since the reviewer was emphasizing the importance of that guard, the right way to honor the comment was to lock the behavior down with a test (there was previously no test for BleedingEdgeToggle at all).

What I added

tests/PHPStan/DependencyInjection/BleedingEdgeToggleTest.php with three cases:

  1. testWithBleedingEdgeRunsCallbackWithToggleSet — the callback observes the toggle set to the requested value while it runs.
  2. testWithBleedingEdgeRestoresPreviousValue — the previous value is restored after the call returns.
  3. testWithBleedingEdgeRejectsGeneratorCallbackAndRestoresValue — a generator callback (i.e. one containing yield) is rejected with ShouldNotHappenException, and the toggle is still restored. This is the exact guard the reviewer flagged: it ensures future tests can't silently reintroduce the race by passing a generator.

Verification

  • New test: 3 tests, 4 assertions, all pass.
  • make phpstan: no errors.
  • make tests: 17439 pass (96 skipped, as usual).

Committed as 523ebc47f and pushed.

@phpstan-bot

Copy link
Copy Markdown
Collaborator Author

I've addressed the reviewer's question. Here's a summary for the PR.

What I did

Yes — the root problem can be fixed by avoiding yield, and I've done that. The three toggle-dependent data providers in ConstantArrayTypeTest are no longer generators:

  • dataAccepts() — builds the unsealed group and the sealed group into two arrays and return [...$unsealed, ...$sealed].
  • dataIsSuperTypeOf() — builds the unsealed object group into $unsealed, the string-based sealedness sets into $sealed, and returns the merge.
  • dataGetArraySize() — accumulates each withBleedingEdge(...) group into $data and returns it.

A returned array runs the provider to completion synchronously, so the global BleedingEdgeToggle is never held across a yield and can never leak into unrelated tests when the provider is abandoned mid-iteration (filtering / random order / early stop). This is exactly the FNSR mid-flight interruption hazard staabm described — it's gone, because there's no suspension point anymore.

One important caveat

I kept the BleedingEdgeToggle::withBleedingEdge(false, ...) wraps, and they're still necessary. Avoiding yield only stops this provider from being a leak source. It does not make object construction deterministic against a toggle that was leaked before the provider runs — e.g. a bleeding-edge container built by an earlier test via ContainerFactory, or another provider in the same worker. That inherited-leak case is exactly what produced the #10/#26/#12/#25 failures, and the explicit withBleedingEdge(false) is what forces those legacy shapes to be built unsealed regardless of ambient state.

So the two mechanisms are orthogonal:

  • No yield → this provider can't cause a leak.
  • withBleedingEdge(false) → this provider is immune to an inherited leak.

The string-based sets in dataIsSuperTypeOf() carry no pre-constructed objects (they're resolved later in the test method under an explicit toggle), so they need no wrap.

Verification

  • ConstantArrayTypeTest + ConstantArrayTypeBuilderTest + IntersectionTypeTest + TypeCombinatorTest + GenericObjectTypeTest under --order-by=random: 1638 pass
  • make tests: 17436 pass (96 skipped)
  • make phpstan: no errors

Committed as 95c610983 and pushed. BleedingEdgeToggle itself is unchanged (the Generator guard in withBleedingEdge now also doubles as a safety net catching any future accidental generator callback).

// data sets must not depend on the ambient global BleedingEdgeToggle: a previously
// created bleeding-edge container in the same worker may have left it set, which
// would seal these shapes at construction time and intermittently flip results.
$unsealed = BleedingEdgeToggle::withBleedingEdge(false, static fn (): array => [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separate bleeding edge true/false into 2 dataproviders

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I addressed the reviewer's request to separate the bleeding edge true/false data sets into 2 data providers.

What changed (tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php)

Each of the three providers that previously merged a withBleedingEdge(false, …) group and a withBleedingEdge(true, …) group into one return is now split into two distinct providers, each stacked onto its test via #[DataProvider]:

  • dataAcceptsdataAcceptsWithoutBleedingEdge + dataAcceptsWithBleedingEdge
  • dataIsSuperTypeOfdataIsSuperTypeOfWithoutBleedingEdge + dataIsSuperTypeOfWithBleedingEdge
  • dataGetArraySizedataGetArraySizeWithoutBleedingEdge + dataGetArraySizeWithBleedingEdge

The …WithoutBleedingEdge providers build under withBleedingEdge(false, …), the …WithBleedingEdge ones under withBleedingEdge(true, …). The core anti-leak guarantees are preserved: each provider still returns a fully-materialized array (no yield), and each still wraps construction in BleedingEdgeToggle::withBleedingEdge(...) so the toggle is restored before returning and immune to an inherited leak.

For dataGetArraySize, the old foreach ([false, true]) loop (which built identical base cases under both toggles, plus an extra false-only group) became the two explicit providers — the false provider keeps the base + extra cases, the true provider keeps the base cases.

Verification

  • ConstantArrayTypeTest: 182 tests pass
  • make phpstan: no errors
  • make tests: 17439 pass (96 skipped, as usual)

Committed as 66c31855b and pushed.

staabm and others added 11 commits June 22, 2026 10:30
…ntArrayTypeTest` data providers

- `dataAccepts()` and `dataGetArraySize()` mutated the global static
  `BleedingEdgeToggle` and kept it mutated across `yield` boundaries. Data
  providers are evaluated lazily and can be abandoned mid-iteration, which left
  the toggle stuck on `true` and leaked into unrelated tests.
- A leaked `true` toggle makes the `ConstantArrayType` objects constructed by
  *other* providers (e.g. `IntersectionTypeTest::dataIsAcceptedBy`) sealed array
  shapes, so `ConstantArrayType::accepts()` returns `No` ("Sealed array shape can
  only accept a constant array") instead of `Maybe` — the intermittent CI failure
  on `IntersectionTypeTest::testIsAcceptedBy` data set phpstan#7.
- Extract `dataWithBleedingEdge(bool, callable)` which sets the toggle, builds the
  toggle-dependent data sets eagerly via the callback, and restores the previous
  value in a `finally` before returning. The objects are therefore constructed
  while the toggle is set, but the global is never observable as mutated once the
  provider yields control.
- Add `testDataProviderDoesNotLeakBleedingEdgeToggle()` which partially consumes
  each bleeding-edge provider and asserts the toggle is restored after every prefix
  length — this reproduces the leak deterministically.
- Audited the rest of the suite: the only `BleedingEdgeToggle` mutations inside
  generator data providers were these two; the `TypeCombinatorTest` /
  `ConstantArrayTypeTest` toggles in test methods already use `try`/`finally` and
  run after providers are evaluated, and the `putenv()` mutations in the error
  formatter tests live in `setUp`/`tearDown`/test methods, not providers.
…set/restore machinery

Centralizes the toggle set/restore logic in a reusable base class:
`withBleedingEdge()` runs a callback under the requested toggle value and
restores the previous value in a `finally`, and `tearDown()` resets the global
default as a safety net even when a partially consumed data provider leaks it.

`ConstantArrayTypeTest` now extends the base class and uses `withBleedingEdge()`
instead of its private `dataWithBleedingEdge()` helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oggle`-dependent tests

`ConstantArrayTypeBuilderTest` and `TypeCombinatorTest` extend the new base class
and replace their inline backup/`setBleedingEdge`/restore blocks with
`withBleedingEdge()`, so the global toggle is never observable as mutated outside
a controlled window.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…est case

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…est providers

The object-based data sets in dataAccepts() and dataIsSuperTypeOf() construct
ConstantArrayType instances at data-provider evaluation time, reading the global
BleedingEdgeToggle to decide default sealedness. They were built under the
ambient toggle and assumed it to be false (legacy, unsealed shapes).

A previously-created bleeding-edge container in the same paratest worker calls
ContainerFactory's BleedingEdgeToggle::setBleedingEdge(true) and never restores
it, leaving the toggle set when these providers are evaluated. The shapes are
then sealed, intermittently flipping results from the expected Yes/Maybe to No
(e.g. array{foo: int} -> isSuperTypeOf(array{foo: int, bar: int})).

Wrap the object-block construction in BleedingEdgeToggle::withBleedingEdge(false,
...) so the shapes are built with an explicit, deterministic toggle value and the
global is restored before yielding, regardless of the leak source.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oviders

dataAccepts(), dataIsSuperTypeOf() and dataGetArraySize() are no longer
generators: they build their toggle-dependent groups into arrays and return
one merged array. A returned array runs the provider to completion
synchronously, so the global BleedingEdgeToggle can never be held across a
`yield` and leak into unrelated tests when the provider is abandoned early.

The explicit BleedingEdgeToggle::withBleedingEdge() wraps stay: they make the
contained ConstantArrayType shapes deterministic regardless of an ambient
toggle leaked from outside the provider (e.g. a bleeding-edge container built
by an earlier test via ContainerFactory). Avoiding `yield` only stops this
provider from being a leak source; it does not protect against an inherited
leaked toggle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Separate the bleeding-edge true/false data sets of dataAccepts,
dataIsSuperTypeOf and dataGetArraySize into two data providers each,
attached to their test via stacked #[DataProvider] attributes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@clxmstaab clxmstaab force-pushed the create-pull-request/patch-m6t0ssr branch from b0f2792 to a85e5ed Compare June 22, 2026 08:30
Comment thread tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php Outdated
The split into with/without providers moved the trailing data sets into
the without-bleeding-edge provider, but they were originally built with
the toggle left on `true` by the preceding loop. Keep them in the
with-bleeding-edge provider so the test data is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php
Revert the split of dataGetArraySize into two providers. Keep the single
provider with its foreach ([false, true]) loop, only converting `yield` to
array accumulation + `return` so the toggle is never held across a `yield`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@VincentLanglet VincentLanglet left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we need that much comments

Comment thread tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php Outdated
Comment thread tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php Outdated
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@staabm staabm changed the title Do not hold the global BleedingEdgeToggle across yield in ConstantArrayTypeTest data providers Do not hold the global BleedingEdgeToggle across yield in ConstantArrayTypeTest data providers - fixes flaky tests Jun 22, 2026
@staabm staabm changed the title Do not hold the global BleedingEdgeToggle across yield in ConstantArrayTypeTest data providers - fixes flaky tests Do not hold the global BleedingEdgeToggle across yield in data providers - fixes racy/flaky tests Jun 22, 2026

@staabm staabm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI still fails with

[2](https://github.com/phpstan/phpstan-src/actions/runs/27944856634/job/82686920896?pr=5911#step:10:333)
2) PHPStan\Type\Constant\ConstantArrayTypeTest::testIsSuperTypeOf with data set #47 ('array{a: int}', 'array{a: int, ...}', PHPStan\TrinaryLogic Object (...))
array{a: int} -> isSuperTypeOf(array{a: int, ...})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Maybe'
+'Yes'

/home/runner/work/phpstan-src/phpstan-src/tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php:1023
/home/runner/work/phpstan-src/phpstan-src/tests/vendor/phpunit/phpunit/src/TextUI/Command.php:146

@phpstan-bot

Copy link
Copy Markdown
Collaborator Author

API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited

@staabm staabm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try again

…peOf

ContainerFactory::create() resets the global BleedingEdgeToggle to the
container's config value. When self::getContainer() built the default
(non-bleeding-edge) container for the first time inside the
withBleedingEdge(true) window, it clobbered the toggle back to false, so the
type strings resolved to legacy (non-sealed) ConstantArrayType shapes and
isSuperTypeOf intermittently returned 'Yes' instead of 'Maybe'.

Fetch the resolver - forcing the container build - before the toggle window,
matching the safe pattern already used in TypeCombinatorTest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@phpstan-bot

Copy link
Copy Markdown
Collaborator Author

Pushed. Here's a summary of what I found and fixed for staabm's review.

Root cause of the remaining #47 flake

The earlier fixes made the object-based data sets immune by building them under withBleedingEdge(false/true) in the providers. But data set #47 is a string-based set in dataIsSuperTypeOfWithBleedingEdge (['array{a: int}', 'array{a: int, ...}', Maybe]), which is resolved at test-method time inside withBleedingEdge(true).

The flake had a different mechanism than the provider-yield leak:

  • ContainerFactory::create() (line 204) calls BleedingEdgeToggle::setBleedingEdge($container's config value) every time a container is built.
  • testIsSuperTypeOf called self::getContainer()->getByType(TypeStringResolver::class) inside the withBleedingEdge(true) callback.
  • When the default (non-bleeding-edge) container happened to be built for the first time inside that window (process/shard/order-dependent — hence intermittent, more on Windows/PHP 7.4/8.0), the build reset the toggle back to false, clobbering the intended true.
  • The string then resolved to a legacy (unsealed === null) ConstantArrayType instead of a sealed one, so array{a: int}array{a: int, ...} returned Yes instead of Maybe.

I proved this directly: building the default container inside the window yields Yes; fetching the resolver before the window yields the correct Maybe.

Fix

Move self::getContainer()->getByType(TypeStringResolver::class) out of the withBleedingEdge(true) callback (7-line diff). This is the exact safe pattern already used by TypeCombinatorTest::testUnion/testIntersect/..., so it makes the two consistent. It's also why those tests never flaked.

Verification

  • make phpstan: no errors
  • make tests: 17442 pass (96 skipped)
  • ConstantArrayTypeTest under --order-by=random: green
  • Mechanism confirmed with before/after reproductions of the container-build clobber

One note for reviewers: the underlying hazard is that ContainerFactory::create() mutates the global toggle. Any future test that builds a container inside a withBleedingEdge window would hit the same trap — the established convention (build the container first) is what avoids it.

BleedingEdgeToggle::setBleedingEdge(true);
try {
$resolver = self::getContainer()->getByType(TypeStringResolver::class);
// Fetch the resolver - and thereby build the container - *before* entering the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shorten the comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I processed this review but have nothing to report.

@staabm staabm merged commit b4df138 into phpstan:2.2.x Jun 22, 2026
390 of 393 checks passed
@staabm staabm deleted the create-pull-request/patch-m6t0ssr branch June 22, 2026 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

intermittent in CI

3 participants