Add execution instrumentation for Karate v2 by daniel-mohedano · Pull Request #11928 · DataDog/dd-trace-java

daniel-mohedano · 2026-07-13T13:44:46Z

What Does This Do

Implements execution instrumentation for the new Karate v2
This instrumentation is responsible for all features related to modifying the execution of tests:
- Early Flake Detection
- Auto Test Retries
- Flaky Test Management Policies

Motivation

Karate v2 is a complete ground-up rewrite of the framework. Because of this, the original karate-1.0 module cannot instrument it. This PR builds upon the changes introduced in #11923

Additional Notes

Most LOC are related to instrumentation tests' span fixtures.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: SDTEST-3816

daniel-mohedano · 2026-07-13T13:49:07Z

@codex review

cit-pr-commenter-54b7da · 2026-07-13T13:59:04Z

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	53.79	55.43	$\color{green}{\blacktriangledown}$ -1.64	54.33	$\color{green}{\blacktriangledown}$ -0.54	76/236
agentEvpProxy	53.81	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

datadog-datadog-us1-prod · 2026-07-13T13:59:13Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 57.03% (+0.00%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: d130935 | Docs | Datadog PR Page | Give us feedback!}

cit-pr-commenter-54b7da · 2026-07-13T14:00:52Z

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	36.82	37.15	$\color{green}{\blacktriangledown}$ -0.33	36.42	$\color{red}{\blacktriangle}$ +0.40	38/115
agentless	37.46	36.42	$\color{red}{\blacktriangle}$ +1.04	36.42	$\color{red}{\blacktriangle}$ +1.04	38/115
agentlessCodeCoverage	45.43	45.38	$\color{red}{\blacktriangle}$ +0.05	44.48	$\color{red}{\blacktriangle}$ +0.95	38/115
agentlessLineCoverage	75.87	74.82	$\color{red}{\blacktriangle}$ +1.05	74.82	$\color{red}{\blacktriangle}$ +1.05	37/114

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-13T14:03:11Z

Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	13.38	9.35	$\color{red}{\blacktriangle}$ +4.03	10.33	$\color{red}{\blacktriangle}$ +3.05	37/115
agentless	8.85	10.13	$\color{green}{\blacktriangledown}$ -1.28	10.13	$\color{green}{\blacktriangledown}$ -1.28	38/115
agentlessCodeCoverage	21.82	16.69	$\color{red}{\blacktriangle}$ +5.13	17.38	$\color{red}{\blacktriangle}$ +4.44	37/113

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-13T14:03:58Z

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	21.11	21.65	$\color{green}{\blacktriangledown}$ -0.54	21.65	$\color{green}{\blacktriangledown}$ -0.54	38/122
agentless	20.11	18.82	$\color{red}{\blacktriangle}$ +1.29	18.82	$\color{red}{\blacktriangle}$ +1.29	37/120
agentlessCodeCoverage	20.37	20.39	$\color{green}{\blacktriangledown}$ -0.02	19.99	$\color{red}{\blacktriangle}$ +0.38	38/119
agentlessLineCoverage	29.85	30.42	$\color{green}{\blacktriangledown}$ -0.57	29.82	$\color{red}{\blacktriangle}$ +0.03	37/118

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-13T14:04:00Z

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	86.88	87.80	$\color{green}{\blacktriangledown}$ -0.92	87.80	$\color{green}{\blacktriangledown}$ -0.92	39/121
agentless	79.60	81.05	$\color{green}{\blacktriangledown}$ -1.45	81.05	$\color{green}{\blacktriangledown}$ -1.45	39/120
agentlessCodeCoverage	96.19	97.04	$\color{green}{\blacktriangledown}$ -0.85	95.12	$\color{red}{\blacktriangle}$ +1.07	39/118
agentlessLineCoverage	111.48	111.62	$\color{green}{\blacktriangledown}$ -0.14	111.62	$\color{green}{\blacktriangledown}$ -0.14	38/117

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-13T14:04:47Z

Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	6.81	9.54	$\color{green}{\blacktriangledown}$ -2.73	9.54	$\color{green}{\blacktriangledown}$ -2.73	35/35

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d130935321

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

dd-octo-sts · 2026-07-13T14:08:16Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.98 s	13.96 s	[-0.8%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent	12.92 s	13.03 s	[-1.5%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent	16.86 s	16.24 s	[-0.6%; +8.3%] (no difference)
startup:petclinic:iast:Agent	16.81 s	16.91 s	[-1.6%; +0.3%] (no difference)
startup:petclinic:profiling:Agent	16.19 s	16.67 s	[-7.2%; +1.5%] (no difference)
startup:petclinic:sca:Agent	16.87 s	16.75 s	[-0.3%; +1.7%] (no difference)
startup:petclinic:tracing:Agent	16.04 s	15.67 s	[-1.9%; +6.7%] (no difference)

Commit: d1309353 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

cit-pr-commenter-54b7da · 2026-07-13T14:13:23Z

Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	94.71	95.12	$\color{green}{\blacktriangledown}$ -0.41	93.23	$\color{red}{\blacktriangle}$ +1.48	42/126
agentless	91.50	89.58	$\color{red}{\blacktriangle}$ +1.92	89.58	$\color{red}{\blacktriangle}$ +1.92	38/121
agentlessCodeCoverage	100.09	99.00	$\color{red}{\blacktriangle}$ +1.09	99.00	$\color{red}{\blacktriangle}$ +1.09	40/120
agentlessLineCoverage	101.17	99.00	$\color{red}{\blacktriangle}$ +2.17	99.00	$\color{red}{\blacktriangle}$ +2.17	38/118

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-13T14:18:09Z

Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	21.48	19.20	$\color{red}{\blacktriangle}$ +2.28	19.20	$\color{red}{\blacktriangle}$ +2.28	40/126
agentless	18.48	18.82	$\color{green}{\blacktriangledown}$ -0.34	18.82	$\color{green}{\blacktriangledown}$ -0.34	38/124
agentlessCodeCoverage	21.63	22.54	$\color{green}{\blacktriangledown}$ -0.91	22.09	$\color{green}{\blacktriangledown}$ -0.46	38/122
agentlessLineCoverage	45.01	44.48	$\color{red}{\blacktriangle}$ +0.53	44.48	$\color{red}{\blacktriangle}$ +0.53	38/127

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-13T14:26:02Z

Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.13	16.36	$\color{green}{\blacktriangledown}$ -0.23	16.04	$\color{red}{\blacktriangle}$ +0.09	38/115
agentless	10.00	9.73	$\color{red}{\blacktriangle}$ +0.27	9.73	$\color{red}{\blacktriangle}$ +0.27	38/116
agentlessCodeCoverage	13.72	13.67	$\color{red}{\blacktriangle}$ +0.05	13.40	$\color{red}{\blacktriangle}$ +0.32	38/114
agentlessLineCoverage	33.52	32.95	$\color{red}{\blacktriangle}$ +0.57	32.95	$\color{red}{\blacktriangle}$ +0.57	37/113

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-13T14:42:23Z

Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	0.65	12.37	$\color{green}{\blacktriangledown}$ -11.72	13.94	$\color{green}{\blacktriangledown}$ -13.29	35/121
agentless	0.49	12.62	$\color{green}{\blacktriangledown}$ -12.13	16.04	$\color{green}{\blacktriangledown}$ -15.55	35/120
agentlessCodeCoverage	63.84	79.45	$\color{green}{\blacktriangledown}$ -15.61	86.07	$\color{green}{\blacktriangledown}$ -22.23	35/120
agentlessLineCoverage	109.31	125.85	$\color{green}{\blacktriangledown}$ -16.54	139.09	$\color{green}{\blacktriangledown}$ -29.78	35/118

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d130935321

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T14:49:16Z

+        }
+
+        // override the return value so the final attempt is the one recorded.
+        result = finalResult;


Propagate retried runtime state to callers

When this runs inside a called feature or callSingle, replacing only the returned ScenarioResult means Karate still keeps the original ScenarioRuntime as FeatureRuntime.lastExecuted after sr.call() returns. Karate's call paths read nestedFr.getLastExecuted().getAllVariables() / propagate config, cookies, and driver from that runtime, so a flaky called scenario that fails once and then passes can report the final retry as successful while returning the first attempt's variables and side effects to the caller. Please make the retry path update the runtime state Karate uses for call result propagation, not just the result object.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-13T14:49:16Z

+          // v2 StepResult is immutable: preserve the error out-of-band, then replace the failing
+          // step with a skipped one so the scenario no longer counts as failed.
+          executionContext.setSuppressedError(stepResult.getError());
+          stepResult = StepResult.skipped(stepResult.getStep(), stepResult.getStartTime());


Preserve @fail semantics when suppressing failures

When a scenario uses Karate's @fail tag and an execution policy suppresses failures (for example quarantine or a suppressed retry attempt), replacing the failed step with a skipped one makes Karate's later applyFailTag() see no failed steps and convert an expected failure into a returned failure. This is separate from the listener-order limitation: the new replacement changes Karate's own final result, so quarantined/managed negative tests can still fail the build even though the expected failure happened.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-13T14:49:16Z

+
+  @Override
+  public boolean isEnabled() {
+    return super.isEnabled() && Config.get().isCiVisibilityExecutionPoliciesEnabled();


Advertise Karate retry capabilities

This enables Karate v2 execution-policy instrumentation, but the handler still starts sessions with KarateUtils.capabilities(), and that method returns Collections.emptyList() in karate-2.0/src/main/java21/.../KarateUtils.java. In normal runs the resulting test spans won't carry the ATR/EFD/quarantine/disabled/attempt-to-fix capability tags that Karate 1.x advertises, so product logic that relies on library capabilities will treat Karate v2 as not supporting the features implemented here.

Useful? React with 👍 / 👎.

datadog-datadog-us1-prod

More details

All critical Karate v2 API contracts verified against the actual karate-core-2.0.9.jar: ScenarioResult.scenario is private final Scenario (ByteBuddy @Advice.FieldValue access works), StepResult.skipped(Step, long) exists, ScenarioRuntime.getScenario()/getFeatureRuntime() are public, and FeatureResult.addScenarioResult() receives the final retry result via the @Advice.Return(readOnly=false) override. The retry loop's CallDepthThreadLocalMap correctly prevents recursive re-entry while allowing called-scenario advice to pass through harmlessly. No behavioral regressions found.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit d130935 · What is Autotest? · Any feedback? Reach out in #autotest}

daniel-mohedano added 2 commits July 13, 2026 14:50

feat: execution instrumentation for karate v2

3d613de

chore: spotless

d130935

daniel-mohedano added type: feature Enhancements and improvements comp: ci visibility Continuous Integration Visibility tag: ai generated Largely based on code generated by an AI or LLM labels Jul 13, 2026

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

daniel-mohedano marked this pull request as ready for review July 13, 2026 14:36

daniel-mohedano requested a review from a team as a code owner July 13, 2026 14:36

daniel-mohedano changed the title ~~Implement execution instrumentation for Karate v2~~ Add execution instrumentation for Karate v2 Jul 13, 2026

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

datadog-datadog-us1-prod Bot reviewed Jul 13, 2026

View reviewed changes

gnufede approved these changes Jul 13, 2026

View reviewed changes

Uh oh!

Conversation

daniel-mohedano commented Jul 13, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

daniel-mohedano commented Jul 13, 2026

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - sbt-scalatest

Uh oh!

datadog-datadog-us1-prod Bot commented Jul 13, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - nebula-release-plugin

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - pass4s

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - reactive-streams-jvm

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - netflix-zuul

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - heliboard

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 13, 2026

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - jolokia

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - okhttp

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - spring_boot

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 13, 2026

Test Environment - sonar-java

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

daniel-mohedano commented Jul 13, 2026 •

edited by atlassian Bot

Loading

datadog-datadog-us1-prod Bot commented Jul 13, 2026 •

edited by datadog-prod-us1-3 Bot

Loading