[#801] Drop the rerunFailingTestsCount setting that never applied to TestNG - #804
Open
vharseko wants to merge 1 commit into
Open
Conversation
…at never applied to TestNG The precommit profile asked failsafe to rerun failing tests three times, but the tests run with the surefire TestNG provider, which does not implement the option: in 3.5.x it is honoured by the JUnit 4/5 providers only, and SUREFIRE-2134 (open since 2022) still tracks adding it. Verified on surefire 3.5.3 and 3.5.6 with a deliberately flaky pair of tests - a plain @test and a @dataProvider driven one: both fail the build on the first occurrence, with no reruns and no Flakes section in the report. So the limitation is not specific to data providers, as suspected in OpenIdentityPlatform#792; the whole setting was a no-op for all 31760 tests of the suite. Remove it and record why in its place, so that a failing precommit run is read as a real failure. Surefire 3.6.0 runs TestNG through the unified JUnit Platform provider and does rerun such methods, but switching providers for this suite is a separate migration. Fixes OpenIdentityPlatform#801
maximthomas
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up of #792 (item 3 of its Proposed fixes).
rerunFailingTestsCount=3was configured for theprecommitprofile(
opendj-server-legacy/pom.xml:1284), but it never rerun anything: these tests are executed bythe surefire TestNG provider, which does not implement the option. Up to and including
3.5.x it is honoured by the JUnit 4/5 providers only, and
SUREFIRE-2134 ("add support for
rerunFailingTestsCount option to TestNG provider", open since December 2022) is still
unresolved.
Contrary to the suspicion in #792, this is not a
@DataProviderspecific limitation - theoption is a no-op for every one of the 31760 tests of the suite.
How it was verified
A minimal project with TestNG 6.14.3,
rerunFailingTestsCount=3and the samereuseForks=false/forkCount=1/parallel=nonesettings as theprecommitprofile, withtwo tests that fail on their first attempt only - a plain
@Testand a@DataProviderdrivenone:
Using auto detected provider org.apache.maven.surefire.testng.TestNGProvider,Tests run: 6, Failures: 2->BUILD FAILURE. No reruns, noFlakes:section, no rerunelements in
TEST-*.xml.surefire-testng-3.5.3.jarclasses never readTestRequest.getRerunFailingTestsCount(), andAbstractSurefireMojo.warnIfRerunClashes()only validates that the value is not negative - nothing warns that the provider ignores it,
which is why the setting looked functional. The
rerun-failing-testspage of the 3.5.3 docssays: "This feature is supported for JUnit 4.x, and (since 3.0.0-M4) JUnit 5.x."
Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProviderTestNG is present. Adding per default org.junit.support:testng-engine:1.1.0, and bothtests are rerun and reported as
Flakes->BUILD SUCCESS. Our TestNG listenerconfiguration (
listener/usedefaultlistenersproperties) survives that switch, andtestng-enginesupports TestNG from 6.14.3, but a rerun re-executes the whole test method(all invocations of its data provider, ~360 for
testSafeDataFromRS), and 3.6.0 is currentlya milestone only. Switching this suite to the unified provider is a migration of its own, not
part of this change.
Change
Remove the setting and put the reasoning in its place, so that a failing
precommitrun isread as a real failure instead of "the retry did not help".
Fixes #801