feat: add mux.cool outbound

[Jolymmiles]

Summary

This PR adds native Xray-compatible mux.cool client and server support for TCP, UDP, and XUDP traffic.

• Implements the Xray Mux wire codec for New, Keep, End, and KeepAlive frames.
• Supports multiplexed TCP streams and UDP/XUDP packet sessions.
• Preserves UDP datagram boundaries and destination addresses.
• Implements XUDP Global ID handling and per-datagram destination metadata.
• Adds full-duplex logical sessions with deadline support and per-stream TCP context cancellation.
• Implements carrier workers and connection pooling.
• Establishes carriers through the base proxy to v1.mux.cool:9527.
• Encodes v1.mux.cool:9527 carriers with the native VLESS Mux command, matching Xray-core in both client/server directions.
• Wraps compatible outbound proxy types generically, including Trojan, VMess, and VLESS.
• Accepts v1.mux.cool carriers through the shared sing-based listener path, including Trojan, VMess, and VLESS listeners.
• Supports a shared primary pool or an optional dedicated UDP/XUDP packet pool.
• Supports active-session and lifetime-session limits per carrier.
• Adds a strict shared limit for physical TCP carriers across the primary and dedicated packet pools.
• Handles carrier reuse, rotation, idle cleanup, failure propagation, and idempotent session termination.
• Adds Xray-compatible UDP/443 handling with reject, allow, and skip policies.
• Rejects configurations that enable both smux and mux.cool.
• Does not silently fall back to the underlying proxy when Mux fails.
• Allows bypassing Mux for UDP/443 only when explicitly configured with skip.
• Optimizes frame encoding, decoding, session lifecycle, carrier writes, and UDP receive paths.
• Adds real mihomo/Xray cross-runtime process tests for TCP, UDP, and XUDP in both directions.

Configuration

mux.cool:
  enabled: true
  max-concurrency: 8
  max-connections: 128
  max-carriers: 4
  xudp-concurrency: 4
  xudp-proxy-udp443: reject

Parameters

enabled

Enables mux.cool for the proxy.

When enabled, logical TCP, UDP, and XUDP sessions can be transported over multiplexed TCP carriers connected through the underlying proxy to v1.mux.cool:9527.

Default: false.

max-concurrency

Maximum number of simultaneously active logical sessions on one primary Mux carrier.

When every existing carrier reaches this limit, another physical carrier is opened. If xudp-concurrency is zero, this limit applies to TCP, UDP, and XUDP sessions in the shared pool.

Default: 8.

A zero or omitted value uses the default.

max-connections

Maximum total number of logical sessions that one carrier may accept during its lifetime.

After a carrier has accepted this many logical sessions, it stops accepting new sessions. Existing sessions continue until completion, while new sessions are assigned to another carrier.

Despite the option name, this is not a limit on the number of physical carrier connections. It is a lifetime logical-session limit for each carrier.

The limit applies independently to carriers in the primary and dedicated packet pools.

Default: 128.

A zero or omitted value uses the default.

max-carriers

Strict maximum number of physical TCP carrier connections owned by this mux.cool wrapper.

The limit is shared by the primary pool and the optional dedicated UDP/XUDP packet pool. In-flight carrier dials count toward the limit, preventing concurrent dial attempts from overshooting it.

When the limit is reached, new logical sessions wait for capacity on an existing carrier or for a physical carrier slot to be released. Waiting respects caller context cancellation and pool shutdown. Mux is not bypassed and the limit is never silently exceeded.

Carrier capacity is released after dial failure, carrier rotation, idle cleanup, carrier failure, or pool shutdown.

Default: 0 (unlimited, preserving the previous behavior).

A negative value is rejected during configuration parsing.

xudp-concurrency

Controls packet-pool isolation and the maximum number of active UDP/XUDP sessions per packet carrier.

• 0: TCP, UDP, and XUDP use the same primary carrier pool.
• Greater than 0: creates a dedicated UDP/XUDP packet pool and uses the configured value as its per-carrier active-session limit.
• Less than 0: rejected as invalid configuration.

A dedicated packet pool prevents long-lived UDP/XUDP sessions from consuming per-carrier TCP session capacity. The implementation routes all packet sessions through this pool when it is enabled, including sessions without an XUDP Global ID. When max-carriers is non-zero, both pools still share the same physical connection budget.

Default: 0.

xudp-proxy-udp443

Controls how UDP traffic targeting port 443, typically QUIC or HTTP/3, is handled.

Supported values:

• reject: reject UDP/443 before opening a carrier. This is the default and matches Xray-core behavior.
• allow: transport UDP/443 through the mux packet path.
• skip: bypass mux.cool and delegate UDP/443 directly to the underlying proxy.

The policy applies only to UDP/443. Other UDP destinations continue through the shared or dedicated packet pool.

Default: reject.

Unknown values are rejected during configuration parsing.

Example Behavior

With the configuration shown above:

• Each primary TCP carrier can have up to 8 active logical sessions.
• UDP/XUDP uses a separate packet carrier pool.
• Each packet carrier can have up to 4 active packet sessions.
• The primary and packet pools can own at most 4 physical TCP carriers in total.
• Every carrier is rotated after accepting 128 logical sessions over its lifetime.
• UDP/443 is rejected.
• Other UDP traffic is transported through the dedicated packet pool.

Protocol Support

The implementation supports:

• Native client and server operation between mihomo instances.
• Bidirectional VLESS interoperability between mihomo and Xray-core clients and servers.
• Generic client-side wrapping of compatible base proxies, including Trojan, VMess, and VLESS.
• Native carrier handling through sing-based server listeners, including Trojan, VMess, and VLESS.
• TCP logical streams.
• UDP packet sessions with preserved datagram boundaries.
• XUDP packet sessions using an 8-byte Global ID.
• Per-datagram target metadata in XUDP Keep frames.
• IPv4, IPv6, and domain destinations.
• Server-first TCP protocols.
• Shared TCP and packet-session carrier reuse.
• Optional dedicated UDP/XUDP packet carrier pools.
• Cross-carrier XUDP rebinding while preserving the Global ID.
• Concurrent stream and packet sessions on the same carrier.

TCP, UDP, and XUDP failures are propagated to the caller without silently bypassing Mux.

The only bypass behavior is the explicit xudp-proxy-udp443: skip policy.

Connection Management

Each carrier tracks:

• Currently active logical sessions.
• Total logical sessions created during its lifetime.
• Session identifiers and routing state.
• Idle state and cleanup timers.
• Read and write failures.

The pool supports:

• Carrier reuse while capacity is available.
• New carrier creation when the concurrency limit is reached.
• Strict global physical-carrier limiting when max-carriers is configured.
• Context-aware waiting when all physical carrier slots are occupied.
• Waking all blocked callers when reusable logical capacity becomes available.
• Carrier rotation when the lifetime-session limit is reached.
• Idle carrier cleanup.
• Context-aware carrier creation.
• Failure propagation to affected logical sessions.
• Idempotent shutdown.
• Closing the primary and dedicated packet pools before the underlying proxy.

When a dedicated packet pool is configured, it has independent workers, concurrency accounting, carrier rotation, and idle cleanup. Its physical carriers still consume the same max-carriers budget as the primary pool.

Error Handling

• Carrier creation errors are returned to the caller.
• Failed carrier dials release their reserved physical-carrier slots.
• Waiting for physical-carrier capacity respects context cancellation and pool shutdown.
• Mux failures do not silently create regular TCP or UDP connections.
• Malformed frames terminate the affected carrier.
• Carrier EOF and write failures are propagated to logical sessions.
• TCP dial-context cancellation closes only the associated logical stream.
• Packet sessions intentionally outlive the context used to create them; their lifetime is controlled through the returned PacketConn, deadlines, carrier failure, or pool shutdown.
• Closing a logical session sends End at most once.
• UDP/443 rejection occurs before carrier creation.
• Explicit skip mode returns the underlying proxy result directly.

Performance

The hot paths were optimized through a bounded 20-pass benchmark- and profile-driven performance cycle.

Each pass started from a benchmark, CPU profile, allocation profile, contention profile, or process-level observation. Only changes with measurable improvements and preserved protocol semantics were retained.

Implemented optimizations:

• Reuses carrier-owned frame encoding buffers.
• Reuses worker-owned metadata decoding buffers.
• Reuses the metadata scratch area for frame-length decoding.
• Uses pooled ownership for internally decoded payload buffers.
• Returns pooled payloads only after TCP or UDP consumers finish using them.
• Keeps the public DecodeFrame ownership model unchanged.
• Uses canonical fast paths for common TCP Keep + Data frames.
• Uses an atomic fast path for carrier closed-state checks.
• Reuses fixed 8 KiB TCP uplink buffers through sync.Pool.
• Avoids context watcher goroutines for non-cancelable contexts.
• Avoids allocating a replacement session map when a carrier is permanently closed.
• Implements direct WaitReadFrom support for packet sessions.
• Avoids the additional UDP staging buffer and copy normally required by the generic packet-connection wrapper.
• Uses structured netip.Addr values instead of repeated IP string conversions.
• Adds a fast path for domain fields containing IP literals.
• Avoids allocations in warmed-up UDP packet writes.
• Preserves structured IP addresses in the internal pooled decoder instead of converting wire IPs to strings and parsing them again.
• Avoids inactive deadline and close-channel select overhead in steady-state UDP/XUDP writes while preserving deadline reset and close-cause behavior.
• Uses atomic read-mostly attachment lookup for server XUDP ingress and egress while retaining generation-safe rebinding.
• Removes a redundant worker availability lock during pool session creation.
• Formats logical TCP remote addresses lazily.
• Uses a shared direct helper for returning internally owned payloads to the pool.
• Keeps primary and dedicated packet pools independently synchronized.
• Adds no limiter work to the default unlimited carrier hot path.
• Uses lost-wakeup-safe broadcast notifications only when a configured physical-carrier limit is saturated.
• Includes persistent benchmarks for codec, worker, session, pool, and packet operations.

Representative Results

Measured on an Apple M3 Max. Results are environment-dependent.

Benchmark	Before	After
Internal pooled frame decode	~996 ns/op, 8192 B/op, 1 alloc	~145 ns/op, 0 B/op, 0 allocs
UDP decode, deliver, and direct receive	~211 ns/op, 96 B/op, 4 allocs	~175 ns/op, 84 B/op, 3 allocs
Client UDP/XUDP packet write	~94 ns/op, 0 B/op, 0 allocs	~52 ns/op, 0 B/op, 0 allocs
Server UDP/XUDP packet write	~76 ns/op, 0 B/op, 0 allocs	~35 ns/op, 0 B/op, 0 allocs
TCP session lifecycle	~5361 ns/op, 10974 B/op, 23 allocs	~4.3 us/op, ~2.8 KiB/op, 20 allocs
Packet-session lifecycle	~698 ns/op, 1360 B/op, 7 allocs	~265 ns/op, 1328 B/op, 6 allocs
Carrier worker write	~141 ns/op	~130 ns/op
Server carrier write	~136 ns/op	~132 ns/op
Carrier shutdown	~88 ns/op, 256 B/op, 3 allocs	~60 ns/op, 208 B/op, 2 allocs
Pooled payload release	~12.3 ns/op	~10.1 ns/op
Pool packet-session churn	~1339 ns/op	~1309 ns/op

The internal pooled decoder is used by carrier workers. The public DecodeFrame function continues to return independently owned payload memory so external callers are not exposed to pooled-buffer lifetime requirements.

Testing

Protocol and Codec Coverage

• Golden wire vectors based on Xray-core revision 0ee156e75c9546a713f6c88c0bd14f5ff953c567.
• New, Keep, End, and KeepAlive frame tests.
• IPv4, IPv6, domain, UDP, and XUDP codec tests.
• XUDP Global ID wire-format tests.
• Per-datagram target metadata tests.
• 8 KiB TCP frame-chunking tests.
• Malformed frame and unexpected EOF tests.
• Public and internally pooled decode-path tests.

Session Behavior

• Full-duplex TCP tests.
• UDP datagram-boundary and destination-preservation tests.
• Server-first protocol tests.
• Read and write deadline tests.
• Context cancellation tests.
• Packet-session dial-context lifetime tests.
• Non-cancelable context lifecycle tests.
• Idempotent-close and single-End tests.
• Carrier EOF and write-failure propagation tests.
• Direct and copied WaitReadFrom behavior tests.
• Pooled payload ownership and consumption tests.

Server Behavior

• Native TCP multiplexing tests through the server runtime.
• UDP packet boundary and target-preservation tests.
• XUDP rebinding across carriers with Global ID preservation.
• Duplicate session ID and per-carrier session-limit rejection tests.
• Authenticated-user isolation for XUDP flows.
• Stale attachment, expiry, and write-failure generation-safety tests.
• Graceful carrier draining during server and listener shutdown.
• Native handler tests through the shared sing-based listener path.
• Listener shutdown tests that verify active mux.cool carriers are drained.

Pool Behavior

• TCP and UDP/XUDP carrier reuse tests.
• Shared TCP and packet-session pool tests.
• Dedicated packet-pool tests.
• Dedicated UDP/XUDP concurrency enforcement tests.
• Active-session limit tests.
• Lifetime-session limit and carrier-rotation tests.
• Strict physical-carrier limit tests under concurrent session creation.
• Shared physical-carrier budget tests across primary and dedicated packet pools.
• Context cancellation while waiting for physical-carrier capacity.
• Capacity release after failed carrier dials, carrier closure, rotation, and shutdown.
• Reuse tests verifying that blocked callers wake when logical capacity becomes available.
• Broadcast wake-up tests for multiple callers waiting on the same saturated carrier.
• Idle-cleanup tests.
• Cross-carrier XUDP rebinding tests.
• Global ID preservation across carrier rebinding.
• Concurrent multi-carrier stress tests.
• Shutdown-order tests for both pools and the underlying proxy.

Configuration and UDP/443 Behavior

• Default and custom configuration parser tests.
• Invalid concurrency and policy validation tests.
• smux and mux.cool conflict tests.
• Default, custom, and invalid max-carriers configuration tests.
• Default UDP/443 rejection tests.
• Explicit UDP/443 allow tests.
• Explicit UDP/443 skip tests.
• Verification that rejection does not open a carrier.
• Verification that skip delegates directly to the base proxy.

Performance Regression Coverage

Persistent benchmarks cover:

• Frame encoding.
• Standard and pooled frame decoding.
• UDP frame decoding.
• Carrier worker writes and shutdown.
• TCP session uplink processing.
• TCP session lifecycle and buffer reuse.
• Packet-session lifecycle.
• Packet-session writes.
• Packet delivery through ReadFrom.
• Direct and copied WaitReadFrom paths.
• Pool packet-session churn.
• Parallel pool packet-session churn.
• Server carrier writes.
• Server UDP/XUDP enqueue and packet-write paths.

Process-Level End-to-End Tests

The integration suite includes a physical-carrier limit test that:

• Builds the current mihomo binary.
• Starts separate mihomo VLESS server and client processes.
• Enables mux.cool with max-concurrency: 8 and max-carriers: 2.
• Places an independent TCP relay between the two cores to count actual accepted physical connections.
• Opens 16 simultaneous logical TCP sessions, filling both carriers.
• Starts 4 additional sessions and verifies that they wait instead of opening a third carrier.
• Releases 4 logical slots and verifies that the blocked sessions complete through carrier reuse.
• Runs a second full wave to verify continued reuse.
• Sends and echoes 36 logical TCP sessions while observing exactly 2 physical carriers and a peak of 2 active physical connections.

Run it with:

go test -tags=integration \
  -run '^TestMuxCoolProcessMaxCarriers$' \
  -v ./transport/muxcool

It also includes a bidirectional cross-runtime compatibility matrix that builds and starts real mihomo and Xray-core processes and verifies:

• mihomo client → Xray server over TCP.
• Xray client → mihomo server over TCP.
• mihomo client → Xray server using normal UDP without an XUDP Global ID.
• Xray client → mihomo server using normal UDP without an XUDP Global ID.
• mihomo client → Xray server using XUDP and a dedicated packet pool.
• Xray client → mihomo server using XUDP and a dedicated packet pool.
• SOCKS5 UDP association, datagram payloads, response destinations, and reverse traffic.
• Native VLESS Mux-command handling for v1.mux.cool carriers.

Normal UDP and XUDP are exercised as distinct wire modes. The normal-UDP scenarios use a zero Global ID, while the XUDP scenarios use source identity and an 8-byte Global ID. Independent TCP, UDP, and DNS echo services verify traffic outside either proxy process.

The test uses XRAY_E2E_BINARY when provided. Otherwise, it builds Xray-core from XRAY_CORE_ROOT or from a sibling Xray-core checkout.

Run the compatibility matrix with:

go test -tags=integration \
  -run '^TestMuxCoolProcessXrayCompatibility$' \
  -v ./transport/muxcool

Validation Results

• Full go test ./... suite passes.
• The affected adapter/outbound and transport/muxcool packages pass under go test -race.
• The complete six-scenario mihomo/Xray process matrix passes under go test -race -tags=integration.
• The cross-runtime matrix passes across five consecutive non-race runs and two consecutive race-detector runs.
• go vet ./adapter/outbound ./transport/muxcool passes.
• git diff --check passes.
• The macOS ARM64 production binary builds and starts successfully.
• The process-level VLESS E2E passes with 36 logical sessions over exactly 2 physical carriers.
• The process-level Xray compatibility matrix passes for TCP, normal UDP, and XUDP in both directions.
• Loopback interface counters remain at zero input errors, output errors, and collisions during the process-level test.
• transport/muxcool statement coverage: 82.9%.

Compatibility

The wire format and XUDP behavior are based on Xray-core revision:

0ee156e75c9546a713f6c88c0bd14f5ff953c567

The feature is configured under the mux.cool proxy key. Mihomo uses kebab-case option names:

mux.cool:
  max-carriers: 4
  xudp-concurrency: 4
  xudp-proxy-udp443: reject

The XUDP options correspond to Xray-core’s xudpConcurrency and xudpProxyUDP443 fields. max-carriers is a mihomo extension that places a strict total cap on physical TCP carriers.

For VLESS, mihomo encodes a carrier targeting v1.mux.cool:9527 with the native VLESS Mux command instead of an ordinary TCP destination request. This matches Xray-core's VLESS behavior and is covered by both unit tests and real-process tests in both client/server directions.

mux.cool is applied as a generic outbound wrapper after the base proxy is created. It therefore works with Trojan as well as VMess, VLESS, and other compatible stream-capable proxy adapters. On the server side, mihomo's Trojan listener uses the same sing-based special-destination handler as VMess and VLESS, so v1.mux.cool carriers are accepted natively.

For example:

proxies:
  - name: trojan-mux
    type: trojan
    server: example.com
    port: 443
    password: password
    sni: example.com
    udp: true
    mux.cool:
      enabled: true
      max-concurrency: 8
      max-connections: 128
      max-carriers: 4
      xudp-concurrency: 4
      xudp-proxy-udp443: reject

Both endpoints must understand mux.cool. A conventional Trojan server without the v1.mux.cool server handler cannot terminate these carriers. The cross-runtime process matrix exercises VLESS against Xray-core for TCP, UDP, and XUDP in both directions. Trojan support is provided by the generic outbound wrapper and the shared Trojan listener handler, but does not yet have a dedicated two-process Trojan E2E test.

Notes

Mux can introduce head-of-line blocking because multiple logical sessions share a TCP carrier. This affects both TCP streams and tunneled UDP/XUDP packets.

A dedicated UDP/XUDP packet pool isolates per-carrier TCP session capacity from long-lived packet sessions. Without max-carriers it may increase the number of physical connections; with max-carriers configured, both pools remain inside the shared cap.

Pooled payload buffers are limited to internal carrier processing and are returned only after their consumer finishes. Public codec callers retain the original independent-memory behavior.

The feature is opt-in and disabled by default. UDP/443 is rejected by default when mux.cool is enabled, matching Xray-core behavior.

[Jolymmiles]

@KT-Yeh could you look pls?

[Jolymmiles]

@wwqgtxx это тоже будет закрыто или хотя бы прочитано описание и код?

[wwqgtxx]

It won't be closed for now, but it won't be merged in the short term either.

The changes in this PR are extensive and clearly largely AI-generated, requiring significant effort to review.

Furthermore, while the feature is interesting, its suitability for merging is open to debate:

The implementation here is heavily biased towards xray, whereas muxcool was actually originated by v2ray-core. Extensions shouldn't focus solely on a single provider; the implications for v2fly-core need to be considered equally.

From another perspective, neither core has seen meaningful modifications to muxcool in years, and its level of refinement lags far behind smux and HTTP/2-based multiplexing. Neither project's newly introduced protocols actually utilize muxcool. We remain unsure whether the ongoing maintenance costs for this feature are justified.

[Jolymmiles]

It won't be closed for now, but it won't be merged in the short term either.

The changes in this PR are extensive and clearly largely AI-generated, requiring significant effort to review.

Furthermore, while the feature is interesting, its suitability for merging is open to debate:

The implementation here is heavily biased towards xray, whereas muxcool was actually originated by v2ray-core. Extensions shouldn't focus solely on a single provider; the implications for v2fly-core need to be considered equally.

From another perspective, neither core has seen meaningful modifications to muxcool in years, and its level of refinement lags far behind smux and HTTP/2-based multiplexing. Neither project's newly introduced protocols actually utilize muxcool. We remain unsure whether the ongoing maintenance costs for this feature are justified.

С учетом того, что в xray никто не трогает mux.cool не думаю, что потребуется вообще, что либо менять, а если возникнут ошибки, я сам посмотрю и подготовлю испралвения. Я понимаю, что есть красивый smux, но к сожалению xray несколько недальновидны и не занимаются повышением качества в данном аспекте