Do not hold the global BleedingEdgeToggle across yield in data providers - fixes racy/flaky tests#5911
Conversation
staabm
left a comment
There was a problem hiding this comment.
there are 2 more failures. are these related and need the same fix?
There were 2 failures:
1) PHPStan\Type\Generic\GenericObjectTypeTest::testIsSuperTypeOf with data set #3 (PHPStan\Type\Generic\GenericObjectType Object (...), PHPStan\Type\Generic\GenericObjectType Object (...), PHPStan\TrinaryLogic Object (...))
ReflectionClass<stdClass> -> isSuperTypeOf(ReflectionClass<object>)
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Maybe'
+'No'
D:\a\phpstan-src\phpstan-src\tests\PHPStan\Type\Generic\GenericObjectTypeTest.php:275
D:\a\phpstan-src\phpstan-src\tests\vendor\phpunit\phpunit\src\TextUI\Command.php:146
2) PHPStan\Type\Generic\GenericObjectTypeTest::testIsSuperTypeOf with data set #2 (PHPStan\Type\Generic\GenericObjectType Object (...), PHPStan\Type\Generic\GenericObjectType Object (...), PHPStan\TrinaryLogic Object (...))
ReflectionClass<object> -> isSuperTypeOf(ReflectionClass<stdClass>)
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'
+'No'
D:\a\phpstan-src\phpstan-src\tests\PHPStan\Type\Generic\GenericObjectTypeTest.php:275
D:\a\phpstan-src\phpstan-src\tests\vendor\phpunit\phpunit\src\TextUI\Command.php:146
|
I've completed a thorough investigation. Here's my evidence-based answer to the reviewer's question. Investigation summaryShort answer: No — those two Why they look similar but aren'tBoth failures show a result collapsing to
The two failing data sets are
Selected per-container by Proof the toggle is not involved
I reproduced the exact scenario under a simulated PHP 8.0 container and toggled the global both ways: And I confirmed per-container isolation holds — building a PHP 8.4 container first does not leak invariant variance into a later PHP 7.4 container in the same process: So this PR's fix is correct and complete for the flakiness it targets; it neither causes nor cures the What the
|
staabm
left a comment
There was a problem hiding this comment.
instead of the current approach, could we extract a new PHPStanBleedingEdgeToggleTestCase class, which implements all necessary set/restore machinery for BleedingEdgeToggle and can be extended by all real use-cases which depend on BleedingEdgeToggle?
|
Done. I addressed the reviewer's request by extracting the set/restore machinery into a reusable base class. What changedNew
Adopted across all real consumers of
The two Verification
Committed as two logical commits (introduce base class + migrate primary file; adopt in remaining files) and pushed. |
|
note to reviewer: there are still other intermittent errors, but these have a different root. will be tackled in a separate PR. |
966d15e to
3d3c222
Compare
staabm
left a comment
There was a problem hiding this comment.
PHP8 tests are still failling
1) PHPStan\Type\Constant\ConstantArrayTypeTest::testIsSuperTypeOf with data set #26 (PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\TrinaryLogic Object (...))
array{a: string} -> isSuperTypeOf(array{a: string, b: int})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'
+'No'
/home/runner/work/phpstan-src/phpstan-src/tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php:1011
/home/runner/work/phpstan-src/phpstan-src/tests/vendor/phpunit/phpunit/src/TextUI/Command.php:146
2) PHPStan\Type\Constant\ConstantArrayTypeTest::testIsSuperTypeOf with data set #10 (PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\TrinaryLogic Object (...))
array{foo: int} -> isSuperTypeOf(array{foo: int, bar: int})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'
+'No'
/home/runner/work/phpstan-src/phpstan-src/tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php:1011
/home/runner/work/phpstan-src/phpstan-src/tests/vendor/phpunit/phpunit/src/TextUI/Command.php:146
3) PHPStan\Type\Constant\ConstantArrayTypeTest::testAccepts with data set #12 (PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\TrinaryLogic Object (...))
array{foo: string} -> accepts(array{foo: string, bar: string})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'
+'No'
/home/runner/work/phpstan-src/phpstan-src/tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php:632
/home/runner/work/phpstan-src/phpstan-src/tests/vendor/phpunit/phpunit/src/TextUI/Command.php:146
4) PHPStan\Type\Constant\ConstantArrayTypeTest::testAccepts with data set #25 (PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\Type\Constant\ConstantArrayType Object (...), PHPStan\TrinaryLogic Object (...))
array{name?: string, color?: string} -> accepts(array{sound: 'test'})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Yes'
|
Done. Here's a summary of what I found and fixed. Root causeThe reviewer's 4 PHP8 failures (
The failing data sets are the object-based sets in I reproduced this exactly: with the toggle forced FixWrapped the object-block construction of both providers in the reviewer-blessed Verification
Committed as One note for the reviewer (not changed here)Because the real leak source is |
I find hacky the need to add use withBleedingEdge(false) for the constantArrayTypeTest. |
my understanding is, that with the FNSR a test which is run from a dataprovider with code which runs btw: as soon as we release PHPStan 3.0, the bleeding edge toggle in the |
I think its just about beeing explicit. after the long block of I like the new code more than the previous one. |
| if ($result instanceof Generator) { | ||
| throw new ShouldNotHappenException('callback is not allowed to yield, to prevent leaking the toggle into unrelated tests.'); | ||
| } |
There was a problem hiding this comment.
note to reviewers: this line is the one which prevents regressions. if we consistently stick with this API, new tests requiring bleeding edge cannot accidentally use generators which can introduce race-conditions for this problem
There was a problem hiding this comment.
Done. The current review was a "note to reviewers" from staabm highlighting that the if ($result instanceof Generator) line in withBleedingEdge is the safety net that prevents the regression class this whole PR addresses — a data provider accidentally holding the toggle across a yield.
Since the reviewer was emphasizing the importance of that guard, the right way to honor the comment was to lock the behavior down with a test (there was previously no test for BleedingEdgeToggle at all).
What I added
tests/PHPStan/DependencyInjection/BleedingEdgeToggleTest.php with three cases:
testWithBleedingEdgeRunsCallbackWithToggleSet— the callback observes the toggle set to the requested value while it runs.testWithBleedingEdgeRestoresPreviousValue— the previous value is restored after the call returns.testWithBleedingEdgeRejectsGeneratorCallbackAndRestoresValue— a generator callback (i.e. one containingyield) is rejected withShouldNotHappenException, and the toggle is still restored. This is the exact guard the reviewer flagged: it ensures future tests can't silently reintroduce the race by passing a generator.
Verification
- New test: 3 tests, 4 assertions, all pass.
make phpstan: no errors.make tests: 17439 pass (96 skipped, as usual).
Committed as 523ebc47f and pushed.
|
I've addressed the reviewer's question. Here's a summary for the PR. What I didYes — the root problem can be fixed by avoiding
A returned array runs the provider to completion synchronously, so the global One important caveatI kept the So the two mechanisms are orthogonal:
The string-based sets in Verification
Committed as |
| // data sets must not depend on the ambient global BleedingEdgeToggle: a previously | ||
| // created bleeding-edge container in the same worker may have left it set, which | ||
| // would seal these shapes at construction time and intermittently flip results. | ||
| $unsealed = BleedingEdgeToggle::withBleedingEdge(false, static fn (): array => [ |
There was a problem hiding this comment.
Separate bleeding edge true/false into 2 dataproviders
There was a problem hiding this comment.
Done. I addressed the reviewer's request to separate the bleeding edge true/false data sets into 2 data providers.
What changed (tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php)
Each of the three providers that previously merged a withBleedingEdge(false, …) group and a withBleedingEdge(true, …) group into one return is now split into two distinct providers, each stacked onto its test via #[DataProvider]:
dataAccepts→dataAcceptsWithoutBleedingEdge+dataAcceptsWithBleedingEdgedataIsSuperTypeOf→dataIsSuperTypeOfWithoutBleedingEdge+dataIsSuperTypeOfWithBleedingEdgedataGetArraySize→dataGetArraySizeWithoutBleedingEdge+dataGetArraySizeWithBleedingEdge
The …WithoutBleedingEdge providers build under withBleedingEdge(false, …), the …WithBleedingEdge ones under withBleedingEdge(true, …). The core anti-leak guarantees are preserved: each provider still returns a fully-materialized array (no yield), and each still wraps construction in BleedingEdgeToggle::withBleedingEdge(...) so the toggle is restored before returning and immune to an inherited leak.
For dataGetArraySize, the old foreach ([false, true]) loop (which built identical base cases under both toggles, plus an extra false-only group) became the two explicit providers — the false provider keeps the base + extra cases, the true provider keeps the base cases.
Verification
ConstantArrayTypeTest: 182 tests passmake phpstan: no errorsmake tests: 17439 pass (96 skipped, as usual)
Committed as 66c31855b and pushed.
…ntArrayTypeTest` data providers
- `dataAccepts()` and `dataGetArraySize()` mutated the global static
`BleedingEdgeToggle` and kept it mutated across `yield` boundaries. Data
providers are evaluated lazily and can be abandoned mid-iteration, which left
the toggle stuck on `true` and leaked into unrelated tests.
- A leaked `true` toggle makes the `ConstantArrayType` objects constructed by
*other* providers (e.g. `IntersectionTypeTest::dataIsAcceptedBy`) sealed array
shapes, so `ConstantArrayType::accepts()` returns `No` ("Sealed array shape can
only accept a constant array") instead of `Maybe` — the intermittent CI failure
on `IntersectionTypeTest::testIsAcceptedBy` data set phpstan#7.
- Extract `dataWithBleedingEdge(bool, callable)` which sets the toggle, builds the
toggle-dependent data sets eagerly via the callback, and restores the previous
value in a `finally` before returning. The objects are therefore constructed
while the toggle is set, but the global is never observable as mutated once the
provider yields control.
- Add `testDataProviderDoesNotLeakBleedingEdgeToggle()` which partially consumes
each bleeding-edge provider and asserts the toggle is restored after every prefix
length — this reproduces the leak deterministically.
- Audited the rest of the suite: the only `BleedingEdgeToggle` mutations inside
generator data providers were these two; the `TypeCombinatorTest` /
`ConstantArrayTypeTest` toggles in test methods already use `try`/`finally` and
run after providers are evaluated, and the `putenv()` mutations in the error
formatter tests live in `setUp`/`tearDown`/test methods, not providers.
…set/restore machinery Centralizes the toggle set/restore logic in a reusable base class: `withBleedingEdge()` runs a callback under the requested toggle value and restores the previous value in a `finally`, and `tearDown()` resets the global default as a safety net even when a partially consumed data provider leaks it. `ConstantArrayTypeTest` now extends the base class and uses `withBleedingEdge()` instead of its private `dataWithBleedingEdge()` helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oggle`-dependent tests `ConstantArrayTypeBuilderTest` and `TypeCombinatorTest` extend the new base class and replace their inline backup/`setBleedingEdge`/restore blocks with `withBleedingEdge()`, so the global toggle is never observable as mutated outside a controlled window. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…est case Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…est providers
The object-based data sets in dataAccepts() and dataIsSuperTypeOf() construct
ConstantArrayType instances at data-provider evaluation time, reading the global
BleedingEdgeToggle to decide default sealedness. They were built under the
ambient toggle and assumed it to be false (legacy, unsealed shapes).
A previously-created bleeding-edge container in the same paratest worker calls
ContainerFactory's BleedingEdgeToggle::setBleedingEdge(true) and never restores
it, leaving the toggle set when these providers are evaluated. The shapes are
then sealed, intermittently flipping results from the expected Yes/Maybe to No
(e.g. array{foo: int} -> isSuperTypeOf(array{foo: int, bar: int})).
Wrap the object-block construction in BleedingEdgeToggle::withBleedingEdge(false,
...) so the shapes are built with an explicit, deterministic toggle value and the
global is restored before yielding, regardless of the leak source.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oviders dataAccepts(), dataIsSuperTypeOf() and dataGetArraySize() are no longer generators: they build their toggle-dependent groups into arrays and return one merged array. A returned array runs the provider to completion synchronously, so the global BleedingEdgeToggle can never be held across a `yield` and leak into unrelated tests when the provider is abandoned early. The explicit BleedingEdgeToggle::withBleedingEdge() wraps stay: they make the contained ConstantArrayType shapes deterministic regardless of an ambient toggle leaked from outside the provider (e.g. a bleeding-edge container built by an earlier test via ContainerFactory). Avoiding `yield` only stops this provider from being a leak source; it does not protect against an inherited leaked toggle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Separate the bleeding-edge true/false data sets of dataAccepts, dataIsSuperTypeOf and dataGetArraySize into two data providers each, attached to their test via stacked #[DataProvider] attributes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
b0f2792 to
a85e5ed
Compare
The split into with/without providers moved the trailing data sets into the without-bleeding-edge provider, but they were originally built with the toggle left on `true` by the preceding loop. Keep them in the with-bleeding-edge provider so the test data is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Revert the split of dataGetArraySize into two providers. Keep the single provider with its foreach ([false, true]) loop, only converting `yield` to array accumulation + `return` so the toggle is never held across a `yield`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
VincentLanglet
left a comment
There was a problem hiding this comment.
Not sure we need that much comments
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BleedingEdgeToggle across yield in ConstantArrayTypeTest data providersBleedingEdgeToggle across yield in ConstantArrayTypeTest data providers - fixes flaky tests
BleedingEdgeToggle across yield in ConstantArrayTypeTest data providers - fixes flaky testsBleedingEdgeToggle across yield in data providers - fixes racy/flaky tests
staabm
left a comment
There was a problem hiding this comment.
CI still fails with
[2](https://github.com/phpstan/phpstan-src/actions/runs/27944856634/job/82686920896?pr=5911#step:10:333)
2) PHPStan\Type\Constant\ConstantArrayTypeTest::testIsSuperTypeOf with data set #47 ('array{a: int}', 'array{a: int, ...}', PHPStan\TrinaryLogic Object (...))
array{a: int} -> isSuperTypeOf(array{a: int, ...})
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Maybe'
+'Yes'
/home/runner/work/phpstan-src/phpstan-src/tests/PHPStan/Type/Constant/ConstantArrayTypeTest.php:1023
/home/runner/work/phpstan-src/phpstan-src/tests/vendor/phpunit/phpunit/src/TextUI/Command.php:146
|
API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited |
…peOf ContainerFactory::create() resets the global BleedingEdgeToggle to the container's config value. When self::getContainer() built the default (non-bleeding-edge) container for the first time inside the withBleedingEdge(true) window, it clobbered the toggle back to false, so the type strings resolved to legacy (non-sealed) ConstantArrayType shapes and isSuperTypeOf intermittently returned 'Yes' instead of 'Maybe'. Fetch the resolver - forcing the container build - before the toggle window, matching the safe pattern already used in TypeCombinatorTest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Pushed. Here's a summary of what I found and fixed for staabm's review. Root cause of the remaining
|
| BleedingEdgeToggle::setBleedingEdge(true); | ||
| try { | ||
| $resolver = self::getContainer()->getByType(TypeStringResolver::class); | ||
| // Fetch the resolver - and thereby build the container - *before* entering the |
There was a problem hiding this comment.
I processed this review but have nothing to report.
Summary
PHPStan\Type\IntersectionTypeTest::testIsAcceptedByfailed intermittently in CI(most often on Windows / PHP 7.4 / 8.0):
The failure is a test-isolation problem, not a type-system bug. The data providers
of
ConstantArrayTypeTestmutate the global staticBleedingEdgeToggleand keepit mutated across
yieldboundaries. Because data providers are evaluated lazily andcan be abandoned mid-iteration (filtering, random ordering, early stop), the global was
left stuck on
trueand leaked into unrelated tests.When the toggle leaks as
true, everyConstantArrayTypeconstructed afterwards — includingthe
array{int, int}built byIntersectionTypeTest::dataIsAcceptedBy— becomes a sealedarray shape.
ConstantArrayType::accepts()then short-circuits a non-constant-array operandwith
No("Sealed array shape can only accept a constant array. Extra keys are not allowed.")instead of the correct
Maybe. Hence the intermittentMaybe→Noflip.The fix makes the providers build their toggle-dependent data sets eagerly and restore the
global before yielding, so the toggle is never observable as mutated outside the provider call.
Changes
tests/PHPStan/Type/Constant/ConstantArrayTypeTest.phpdataWithBleedingEdge(bool $bleedingEdge, callable $cases): array— sets the toggle,invokes the callback to build the data sets while the toggle is set, and restores the
previous value in a
finallybefore returning.dataAccepts()to produce its two bleeding-edge-dependent groups (unknown-sealednessand sealed) through
dataWithBleedingEdge(...)+yield from, instead of toggling the globalinline between
yields.dataGetArraySize()the same way (itsforeach ([false, true] ...)loop previouslyheld the toggle across every
yield, and even left it ontruefor the trailing group).testDataProviderDoesNotLeakBleedingEdgeToggle()(data-provided over both providers)that partially consumes each provider for every prefix length and asserts the toggle is
restored — a deterministic reproduction of the flaky behavior.
Root cause
Pattern: a lazily-evaluated generator data provider mutates a global static and holds it
mutated across a
yield. When such a generator is abandoned before completion (or itspost-last-
yieldcleanup never runs), the global stays mutated for the rest of the process,and every later object construction that reads that global is silently affected.
Concretely:
ConstantArrayType::__construct()readsBleedingEdgeToggle::isBleedingEdge()to decide thedefault sealedness (
unsealed = [never, never]when bleeding edge is on).ConstantArrayTypeTest::dataAccepts()anddataGetArraySize()flipped the toggle totrueand yielded objects while it was held, restoring it only after the final
yield.left the global on
true, soIntersectionTypeTest::dataIsAcceptedBybuilt sealed arrays andtestIsAcceptedBydata set DOMNode child and sibling accessor properties should be nullable #7 returnedNo.The fix removes the only two places where this pattern occurred. The object construction still
happens under the requested toggle value (the callback runs inside the set/restore window), but
the global is restored synchronously before any value is yielded, so no consumer can ever observe
the mutated state.
Test
testDataProviderDoesNotLeakBleedingEdgeToggle()consumesdataAccepts()anddataGetArraySize()for every prefix length 1..60 and assertsBleedingEdgeToggle::isBleedingEdge()is
falseafterwards. Verified it fails before the fix (toggle leakstrueafter consuming6 data sets) and passes after.
IntersectionTypeTest,ConstantArrayTypeTest, andTypeCombinatorTestpass under repeated--order-by=randomruns; the full suite (make tests) andmake phpstanare green.Analogous cases probed
(
BleedingEdgeToggle,ini_set,putenv,setlocale, static setters held acrossyield):the only leaking generators were these two providers.
TypeCombinatorTest::testUnion/testIntersect/...andConstantArrayTypeTest::testIsSuperTypeOf/testSealedness/testEqualsTreatsLegacyNullAndSealedMarkerAsEqualtoggle the global inside testmethods with
try/finally, and run after providers are evaluated — they cannot contaminateprovider-time construction and were left unchanged.
putenv()calls in the error-formatter tests live insetUp/tearDown/test methods, notproviders, and are restored in
tearDown.Fixes phpstan/phpstan#14855