Skip to content

[#801] Drop the rerunFailingTestsCount setting that never applied to TestNG - #804

Open
vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:issues/801-drop-rerun-failing-tests-count
Open

[#801] Drop the rerunFailingTestsCount setting that never applied to TestNG#804
vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:issues/801-drop-rerun-failing-tests-count

Conversation

@vharseko

Copy link
Copy Markdown
Member

Follow-up of #792 (item 3 of its Proposed fixes).

rerunFailingTestsCount=3 was configured for the precommit profile
(opendj-server-legacy/pom.xml:1284), but it never rerun anything: these tests are executed by
the surefire TestNG provider, which does not implement the option. Up to and including
3.5.x it is honoured by the JUnit 4/5 providers only, and
SUREFIRE-2134 ("add support for
rerunFailingTestsCount option to TestNG provider", open since December 2022) is still
unresolved.

Contrary to the suspicion in #792, this is not a @DataProvider specific limitation - the
option is a no-op for every one of the 31760 tests of the suite.

How it was verified

A minimal project with TestNG 6.14.3, rerunFailingTestsCount=3 and the same
reuseForks=false / forkCount=1 / parallel=none settings as the precommit profile, with
two tests that fail on their first attempt only - a plain @Test and a @DataProvider driven
one:

  • surefire 3.5.3 (our version) and 3.5.6 (latest stable):
    Using auto detected provider org.apache.maven.surefire.testng.TestNGProvider,
    Tests run: 6, Failures: 2 -> BUILD FAILURE. No reruns, no Flakes: section, no rerun
    elements in TEST-*.xml.
  • Cross-checks: the surefire-testng-3.5.3.jar classes never read
    TestRequest.getRerunFailingTestsCount(), and AbstractSurefireMojo.warnIfRerunClashes()
    only validates that the value is not negative - nothing warns that the provider ignores it,
    which is why the setting looked functional. The rerun-failing-tests page of the 3.5.3 docs
    says: "This feature is supported for JUnit 4.x, and (since 3.0.0-M4) JUnit 5.x."
  • surefire 3.6.0-M1 for comparison:
    Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
    • TestNG is present. Adding per default org.junit.support:testng-engine:1.1.0, and both
      tests are rerun and reported as Flakes -> BUILD SUCCESS. Our TestNG listener
      configuration (listener / usedefaultlisteners properties) survives that switch, and
      testng-engine supports TestNG from 6.14.3, but a rerun re-executes the whole test method
      (all invocations of its data provider, ~360 for testSafeDataFromRS), and 3.6.0 is currently
      a milestone only. Switching this suite to the unified provider is a migration of its own, not
      part of this change.

Change

Remove the setting and put the reasoning in its place, so that a failing precommit run is
read as a real failure instead of "the retry did not help".

Fixes #801

…at never applied to TestNG

The precommit profile asked failsafe to rerun failing tests three times, but the
tests run with the surefire TestNG provider, which does not implement the option:
in 3.5.x it is honoured by the JUnit 4/5 providers only, and SUREFIRE-2134 (open
since 2022) still tracks adding it. Verified on surefire 3.5.3 and 3.5.6 with a
deliberately flaky pair of tests - a plain @test and a @dataProvider driven one:
both fail the build on the first occurrence, with no reruns and no Flakes section
in the report. So the limitation is not specific to data providers, as suspected
in OpenIdentityPlatform#792; the whole setting was a no-op for all 31760 tests of the suite.

Remove it and record why in its place, so that a failing precommit run is read as
a real failure. Surefire 3.6.0 runs TestNG through the unified JUnit Platform
provider and does rerun such methods, but switching providers for this suite is a
separate migration.

Fixes OpenIdentityPlatform#801
@vharseko
vharseko requested a review from maximthomas July 31, 2026 11:37
@vharseko vharseko added CI build tests Test suites: fixing, enabling, un-disabling labels Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build CI tests Test suites: fixing, enabling, un-disabling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Confirm whether rerunFailingTestsCount applies to TestNG @DataProvider driven methods

2 participants