[2/3] Integrate XSK maps into the core XDP datapath by ProjectsByJackHe · Pull Request #6003 · microsoft/msquic

ProjectsByJackHe · 2026-05-18T02:47:11Z

Description

Full E2E demo using tools from this PR : DEMO

Part 2 of the plan: #5982
Fixes #5972

Adds the core msquic datapath integrations for the new API contract defined in #5983

Key design decisions:

If map mode is enabled, then we do not create the WinSock or Epoll or .... datapaths. No more best effort!
Init failures in map mode hard fails the datapath initialization. No silent failures!

Testing

Added datapath unit tests. See the E2E demo.

Additionally, the plan is to leverage some supporting PRs that included code to ingest the latest XDP version, and kicking off a couple of manual CI runs based on that: https://github.com/microsoft/msquic/actions/runs/27313685557/job/80690420168

Documentation

Will come as part 3 in the plan: #5982

codecov · 2026-05-18T03:39:00Z

Codecov Report

❌ Patch coverage is 60.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.51%. Comparing base (f87bfd3) to head (3432322).
⚠️ Report is 19 commits behind head on main.

Files with missing lines	Patch %	Lines
src/core/settings.c	33.33%	2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (60.00%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6003      +/-   ##
==========================================
+ Coverage   84.88%   85.51%   +0.62%     
==========================================
  Files          60       60              
  Lines       18797    18846      +49     
==========================================
+ Hits        15956    16116     +160     
+ Misses       2841     2730     -111

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ProjectsByJackHe · 2026-05-19T00:10:48Z

Planning on waiting until #6006 gets resolved to address the next wave of feedback.

mtfriesen · 2026-05-20T19:01:12Z

It's not clear to me why we don't have end-to-end tests exercising everything from providing the XSKMAP before creating a registration through actual XDP data paths flowing? Since we intend to officially support this feature, it is risky for both us and our partners to supply/depend on an API that does not have regression test coverage and a paved pattern to follow.

ProjectsByJackHe · 2026-05-20T19:34:19Z

It's not clear to me why we don't have end-to-end tests exercising everything from providing the XSKMAP before creating a registration through actual XDP data paths flowing? Since we intend to officially support this feature, it is risky for both us and our partners to supply/depend on an API that does not have regression test coverage and a paved pattern to follow.

That's in the works. I took the approach of building up an E2E flow manually first to see the shape of how the datapath integration would look like. There's likely going to be some limitations in the CI environment that prevents some scenarios from being exercised, but I am currently adding coverage for the parts that can be automated now that the skeleton of the general changes is mapped out in the current PR iteration.

mtfriesen · 2026-05-20T20:56:59Z

Let's ensure we add tests with each chunk of work we do. I understand it may be a lot of work to add tests, but the best time to protect the code with automation and review the coverage is when making the change.

tests all working now

ProjectsByJackHe · 2026-05-26T22:02:08Z

Let's ensure we add tests with each chunk of work we do. I understand it may be a lot of work to add tests, but the best time to protect the code with automation and review the coverage is when making the change.

XDP needs to release a new driver version consumable by MsQuic in the CI before I can push my tests.

I have them right now working over a local VM with the latest xdp drivers installed, and find that everything is working.

ProjectsByJackHe · 2026-05-28T23:10:32Z

Either we merge #6034 (or wait until the next XDP release) to add automated coverage, or having a one-off manual dispatch for the tests + manual local VM runs is good enough to give confidence to merge this integration. Let's decide on this first and align timelines. Currently, the map tests are pushed for review, but disabled.

mtfriesen · 2026-06-11T19:43:16Z

+
+_IRQL_requires_max_(PASSIVE_LEVEL)
+QUIC_STATUS
+CxPlatDpRawInsertXskByMapConfigs(


Is there a corresponding cleanup?

are you asking where in msquic we close/cleanup the map handles given to msquic? If so, the ownership of the map handles is external; the app should own creation/deletion. When XSKs get destroyed upon msquic cleanup, it is the responsibility of the app to handle cleaning up the map handles as well, otherwise it will just have a bunch of stale entries in the map, where xdp will be directing traffic to dead sockets (dropping them basically).

When XSKs get destroyed upon msquic cleanup, it is the responsibility of the app to handle cleaning up the map handles as well

I'm not sure I follow this - usually symmetry for create/delete is preferred. If MsQuic is inserting entries into the maps, it seems ideal for it to revert that during its teardown.

+1 to this.
We recently had to do something similar for rules. We should have a best effort removal from the map on cleanup and failure.
The app owns the map, but MsQuic owns the XSK, so lets make sure we don't leave them in the map.

ProjectsByJackHe · 2026-06-12T01:38:29Z

+    }
+
+    const uint32_t FakeIfIndex = 0xDEAD;
+    const QUIC_XDP_MAP_HANDLE FakeHandle = (QUIC_XDP_MAP_HANDLE)(intptr_t)-1;


Might be wondering, why -1? it was previously 0x1234 (which I thought was out of bounds for windows handles).
Why not NULL or INVALID_HANDLE (0)? Indeed, 0 is invalid on windows but not posix.

Given that the general abstraction of raw-only-datapath mode is cross-platform, POSIX runs these tests too, and so we can add that coverage.

That's a good point. Maybe we need a cross-plat abstraction for "invalid [socket|map|file] handle" which can either be a value we know must be invalid at compile time, or open some dynamically created object (e.g., an event object) that we expect will mismatch the expected handle type.

mtfriesen

We need to solve the testing problem - map mode should run all existing MsQuic tests that are not intrinsically incompatible with an XDP map.

Nearly all MsQuic protocol level tests should be compatible with XDP maps, so getting those exercised is a min bar.

guhetier · 2026-06-12T16:32:01Z

    CxPlatInitialize();

-    CXPLAT_DATAPATH* Datapath;
+    CXPLAT_DATAPATH* Datapath = NULL;


nitnit: this is C++, nullptr or default init should be prefered.

Suggested change

CXPLAT_DATAPATH* Datapath = NULL;

CXPLAT_DATAPATH* Datapath{};

guhetier · 2026-06-12T16:50:37Z

+    if (Source->IsSet.XdpEnabled && !Source->XdpEnabled && MsQuicLib.XdpMapConfigCount > 0) {
+        QuicTraceLogError(
+            SettingXdpDisabledInMapMode,
+            "[ lib] Error: XdpEnabled cannot be set to FALSE when XDP map mode is active.");


nit: let's make logs slightly more actionable for users. "xdp map mode" is our current name for this feature, but we don't expose it in the API. Let's also avoid double negatives.

Suggested change

"[ lib] Error: XdpEnabled cannot be set to FALSE when XDP map mode is active.");

"[ lib] Error: Xdp must be enabled when an XDP map was configured.");

guhetier · 2026-06-12T16:57:05Z

+    //
+    // N.B. Currently only supported for Windows user-mode.
+    //
+    const struct QUIC_XDP_MAP_CONFIG* XdpMapConfigs;


nit: consider SAL to annotate the length.

guhetier · 2026-06-12T16:58:38Z

+    //
+    // N.B. Currently only supported for Windows user-mode.
+    //
+    const struct QUIC_XDP_MAP_CONFIG* XdpMapConfigs;


style: We don't repeat the "struct" keyword on use:

Suggested change

const struct QUIC_XDP_MAP_CONFIG* XdpMapConfigs;

const QUIC_XDP_MAP_CONFIG* XdpMapConfigs;

guhetier · 2026-06-12T17:01:16Z

+    // The map configs must remain valid for the lifetime of the datapath.
+    //
+    // N.B. Currently only supported for Windows user-mode.
+    //


nit: This comment is a bit overly detailed for the config parameter; it is likely to fall out of sync.
The details would be better in a design doc file or at the relevant point in the code.

Suggested change

//

//

// External XDP map configurations. When present, the datapath insert XSK sockets in

// the provided maps at instead of configuring per-connection rules.

// The map configs must remain valid for the lifetime of the datapath.

//

// N.B. Currently only supported for Windows user-mode.

//

guhetier · 2026-06-12T19:13:08Z

+        // since we are skipping OS platform specific initializations.
+        //
+        CXPLAT_DBG_ASSERT(InitConfig->XdpMapConfigs != NULL);
+        if (NewDataPath == NULL) {


Already checked above?

guhetier · 2026-06-12T19:16:44Z

+        if (NewDataPath == NULL) {
+            return QUIC_STATUS_INVALID_PARAMETER;
+        }
+        if (UdpCallbacks != NULL) {


if UdpCallbacks == NULL, shouldn't we fail? Or is it intentional to support a scenario with no callbacks?
I suspect this is an artifact of the "TCP or UDP callbacks are needed" logic, but we only have UDP here.

guhetier · 2026-06-12T19:19:27Z

-        DataPathInitialize(
+    if (InitConfig->XdpMapConfigCount > 0) {
+        //
+        // Raw-only datapath: the raw datapath must initialize successfully


The logic in that first if branch would be better factored in a function.
Either having a RawOnlyDataPathInitialize function, or extracting the logic from DataPathInitialize that is always needed, so you can do a DataPathCommonInitialize followed by RawDataPathInitialize here.

CxPlatDataPathInitialize should only deal with dispatching, not have low level logic.

guhetier · 2026-06-12T19:55:34Z


    CXPLAT_DBG_ASSERT(Datapath->UdpHandlers.Receive != NULL || Config->Flags & CXPLAT_SOCKET_FLAG_PCP);
-    CXPLAT_DBG_ASSERT(IsServerSocket || Config->PartitionIndex < Datapath->PartitionCount);
+    CXPLAT_DBG_ASSERT(CxPlatDpRawIsRawDatapathOnly(Datapath->RawDataPath) || IsServerSocket || Config->PartitionIndex < Datapath->PartitionCount);


Question: Why do we need this change? It isn't clear to me why xdp maps would interfere with partioning.

guhetier · 2026-06-12T20:11:58Z

+
+_IRQL_requires_max_(PASSIVE_LEVEL)
+QUIC_STATUS
+CxPlatDpRawInsertXskByMapConfigs(


+1 to this.
We recently had to do something similar for rules. We should have a best effort removal from the map on cleanup and failure.
The app owns the map, but MsQuic owns the XSK, so lets make sure we don't leave them in the map.

ProjectsByJackHe added 13 commits May 14, 2026 17:46

add tools

0020948

phase 2 looks good so far

9830250

phase 2 improvements

7ba0bf1

better diagnostics

b364f47

phase 3 plumbing...

01df062

phase 3: iter 2

3097b18

bug fixes man

5750131

E2E

bbf64e0

fix comment

c62d7b1

update xdp commit

08ff04e

get rid of tools

b8285dc

get rid of preview features

4546dfe

add unit tests

ba539a7

ProjectsByJackHe requested a review from a team as a code owner May 18, 2026 02:47

ProjectsByJackHe mentioned this pull request May 18, 2026

[2/3] Add CIBIR, XDP, and XSK maps support to the sample tools #5999

Open

CLOG

d15f3a7

guhetier reviewed May 18, 2026

View reviewed changes

fix abi incompat

4fe8094

ProjectsByJackHe added 2 commits May 19, 2026 14:29

address code feedback

47f162d

Merge branch 'main' into jackhe/xdp_map_dp_int_pr_2_core

0b83796

mtfriesen reviewed May 20, 2026

View reviewed changes

Comment thread src/platform/datapath_raw_xdp_win.c Outdated

mtfriesen reviewed May 20, 2026

View reviewed changes

Comment thread src/platform/datapath_raw_xdp_win.c Outdated

ProjectsByJackHe added 3 commits May 20, 2026 14:36

fix dp tests + update logging

f68a1d8

infer xdp enablement if map mode is set

d8fe2d2

set xdpenabled to false explicitly in map mode returns error

96ded75

ProjectsByJackHe added 6 commits May 22, 2026 15:39

CLOG

4eb0d70

remove bogus test

c9e7b1f

stash changes

c1fa28c

tests all working now

improve tests

3b85f5e

add cibir/qtip matrix

815ff66

add CI to target map mode explicitly

39ce0e6

ProjectsByJackHe added 2 commits May 28, 2026 16:34

comment out map job

3d9d478

Merge branch 'main' into jackhe/xdp_map_dp_int_pr_2_core

9716cef

ProjectsByJackHe commented May 28, 2026

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

guhetier requested changes May 29, 2026

View reviewed changes