configfailurepolicy=continue only works for BeforeTest when using TestNG XML file

[bj-9527]

configfailurepolicy=continue only works for BeforeTest when using TestNG XML file

Fixes #2731

Did you remember to?

• Add test case(s)
• Update CHANGES.txt
• Auto applied styling via ./gradlew autostyleApply

We encourage pull requests that:

• Add new features to TestNG (or)
• Fix bugs in TestNG

If your pull request involves fixing SonarQube issues then we would suggest that you please discuss this with the
TestNG-dev before you spend time working on it.

Note: For more information on contribution guidelines please make sure you refer our Contributing section for detailed set of steps.

[krmahadevan]

This is where I have a concern.

The documentation says this Whether TestNG should continue to execute the remaining tests in the suite or skip them if an @Before* method fails. Default behavior is skip.

But here we are trying to move forward with trying to run the next lower level configuration even when the earlier higher order one failed.

[bj-9527]

Whether TestNG should continue to execute the remaining tests in the suite or skip them if an @before* method fails. Default behavior is skip.

In my opinion, the default behavior is skip because of that the default config policy is set to FailurePolicy DEFAULT_CONFIG_FAILURE_POLICY = FailurePolicy.SKIP; , I'm not sure the description for Continue, will look up in document.

[juherr]

I think the behavior change should be configurable because there are too many changes that will be difficult to review (so, right by default).

[bj-9527]

I think the behavior change should be configurable because there are too many changes that will be difficult to review (so, right by default).

I added two test class, one for skip and anther is for continue, please help to check

[juherr]

The tests are failling and the change must not modify the current tests or the behavior by default. The new behavior may enabled by a new option.

[juherr]

(Written by Claude, on Julien's behalf.)

@bj-9527 — thank you for this, and sorry it sat unanswered for so long. Your reading of the issue was right about the symptom: on current master, continue really does let the test run for a failed @BeforeTest and for nothing else. I measured the full matrix and posted it on #2731.

I am closing this one, though, and not because it needs a rebase.

The two conditions here (&& !m_continueOnFailedConfiguration in ConfigInvoker, && getConfigFailurePolicy() == SKIP in TestInvoker) collapse continue into "never skip anything, ever". But continue already has a meaning for @BeforeClass and @BeforeMethod: it narrows what a configuration failure invalidates — to the failing instance, to the failing invocation — instead of poisoning the whole class. That is what GITHUB-990, GITHUB-2148 and GITHUB-2862 (ignoreFailure) were built on, and removing it would regress all three.

The test changes show the size of that: FailurePolicyTest had to be flipped to run under skip, its continue cases moved into a new file with all-zero skip counts, and test/listeners/issue1777/IssueTest changed from testSkipped_test_method: test1 to testSuccess_test_method: test1 — i.e. a test whose @BeforeMethod threw would report success. That is a breaking change to the semantics rather than a fix, and it is the concern @krmahadevan raised on #2731 and @juherr raised in review here.

There is also a real design question underneath, still unanswered: is @BeforeTest + continue the bug (too permissive, should skip like @BeforeClass) or the model (and the other levels are too strict)? Either answer breaks users, so it needs a maintainer decision before any code moves.

What has happened instead, building directly on what you found here:

• Pin and document what configfailurepolicy actually does #3426 — a characterization test pinning the whole 4×2 matrix, so the inconsistency is visible in the suite rather than only in an issue, plus javadoc that stops XmlSuite.FailurePolicy from being silent about its own semantics.
• Say what configfailurepolicy actually does testng-team.github.io#85 — the website row that promised the behaviour TestNG does not have, which is what set this expectation in the first place.

#2731 stays open for the design call. If it lands on "@BeforeTest should be permissive everywhere", your change is the shape it would take, and I would rather revisit it then with the tests already in place than merge it now against expectations nobody has agreed on.