Report failed flaky tests as skip#11931
Conversation
PerfectSlayer
requested review from
randomanderson
and removed request for
a team
PerfectSlayer
added
tag: no release notes
PerfectSlayer
added
the
tag: ai generated
![datadog-official[bot]](https://avatars.githubusercontent.com/in/1358509?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The PR safely implements downgrading failed flaky tests to "skip" status when continueOnFailure=true, maintaining proper ordering and guard conditions to prevent double-tagging. Code is well-documented, handles the environment variable safely, and preserves existing behavior for normal test runs.
📊 Validated against 3 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit 7d30705 · What is Autotest? · Any feedback? Reach out in #autotest
PerfectSlayer
changed the title
|
🎯 Code Coverage (details) 🔗 Commit SHA: 94dbdf3 | Docs | Datadog PR Page | Give us feedback! |
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
| /// {@link #tagSyntheticFailures()} are left untouched. | ||
| void tagFailuresAsSkipped() { | ||
| for (var testcase : testcases()) { | ||
| if (hasFinalStatusProperty(testcase) || !isFailed(testcase)) { |
There was a problem hiding this comment.
From a semantic point of view, we should treat all tests in that job as final_status=skip. The reason is that this status is really meant to describe the impact of the CI, regardless of its status.
For green CI, it does not change anything, as only test failures are looked. But the day we'll start to look over the test based to check tests that are ignored while been always green, it'll be be the canonical way of looking them : status=pass AND final_status=skip.
Though, as the present ticket is about the impact on Green CI, feel free to ignore my comment 😃 .
There was a problem hiding this comment.
Let’s get it right now, it will save us time late.
I pushed follow up changes: 94dbdf3
There was a problem hiding this comment.
we should treat all tests in that job as
final_status=skip
Including normally passing tests?
From what I understand here you're saying it should behave the way CI is behaving: the ENV is to "continue on failure", so that if a failure halts CI, all previously passing tests would report as skipped? Correct?
I don't quite follow your second statement/comment. Why would we want to look at passed tests that have final_status=skip except to exclude such cases from SLAs (false positives)?
There was a problem hiding this comment.
I see Bruce already addressed the first part to report them all as skipped 94dbdf3
Edit: My webpage was stale so this comment came in late 😄
There was a problem hiding this comment.
@cbeauchesne Is there a way to check the results are following your expectations or do I need to merge it first?
4 participants
What Does This Do
This PR updates the test result collector to mark the failed flaky tests as skip.
Motivation
Avoid reporting "failures" to the Test Optimization product.
Additional Notes
It additionally fixes a typo for the release notes tag documentation.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: APMLP-1267