Skip to content

Test writer backpressure and fix drain cancellation and shutdown - #112

Merged
egorsmkv merged 4 commits into
masterfrom
test/writer-backpressure-tdd
Oct 10, 2026
Merged

egorsmkv merged 4 commits into
masterfrom
test/writer-backpressure-tdd

Conversation

@egorsmkv

@egorsmkv egorsmkv commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Native writers had two drain lifecycle gaps: cancelled waits were retained for as long as a peer kept writing paused, and drain() on a closing transport returned success without letting connection_lost run. Retried timeouts could accumulate waiter objects; a write/drain loop could miss shutdown.

This separate test-first PR is based on master 3ff820a and is independent of the reader-usecase PR #111.

  • The first commit adds real TCP writer regressions. Before fixes: 3 failed, 12 passed; all failures were rsloop cases. asyncio and uvloop passed.
  • The second commit removes completed drain waiters by identity using a weak protocol callback, and yields on the cold closing path before checking drain state. Reader errors retain precedence. Ready unpaused drains continue to reuse their cached Future without allocating a callback.
  • The expanded matrix has 30 cases, covering cancellation and timeout churn through both writer and protocol APIs, simultaneous drains, slow-peer backpressure, ordered close/half-close delivery, half-close replies, close/abort wakeups, repeated drains and error precedence.

Tests use real loopback TCP, events, bounded waits and explicit task/socket cleanup. A raw peer controls consumption; deterministic protocol pause callbacks test retention without kernel timing assumptions. Weak references detect retained cancelled Futures while the peer is still paused. Loop exceptions and leaked tasks fail the tests. Optional uvloop cases skip where unavailable; rsloop cases assert the native writer is used.

Validation actually executed locally on Linux:

  • CPython 3.14.7 release extension: 30/30 new cases passed across asyncio, rsloop and uvloop.
  • Full Python suite with private Redis/hiredis: 511 passed, 2 known Unix-socket EPERM failures, 3 skipped, 98 tooling cases deselected.
  • Tooling separately: 98 passed.
  • CPython 3.15.0rc3 free-threaded, GIL disabled: writer lifecycle plus stream cancellation/connection/TLS lifecycle 144 passed, 70 skipped; uvloop unavailable.
  • Rust default suite: 331 passed, 15 EPERM failures from unavailable io_uring/Unix-socket operations.
  • All-target/all-feature Clippy with warnings denied, Rust formatting, changed Python Ruff/Pyright, strict MkDocs: passed.
  • Hotpath inventory: 2081 definitions, zero missing hooks.

CI exposed an overstrict assertion in the new TCP test on Windows Proactor: other drains can legitimately finish while one cancellation is being delivered, because the kernel may accept the queued bytes before the peer application reads. The TCP test now checks that other drains are not cancelled and still deliver every byte; a separate explicitly paused protocol test enforces that cancellation does not wake neighbouring waiters. The test also uses asyncio.TimeoutError for Python 3.10 compatibility. After these test fixes, local release: 30 passed; free-threaded writer matrix: 20 passed, 10 skipped.

Local full suites are not claimed green. Final-head CI (25f3d3e) completed successfully: Tests run 38046594215 and MkDocs run 38046594204. The full Linux/macOS/Windows Python matrix, including free-threaded builds, passed, alongside default/all-feature Rust, Clippy, Miri, profiling/hotpath, tooling, AnyIO and Redis/hiredis. Windows/macOS validation was performed by GitHub Actions, not locally. No final CI failures remain. No throughput improvement is claimed.

@egorsmkv
egorsmkv merged commit 36a60e2 into master Oct 10, 2026
40 checks passed
@egorsmkv
egorsmkv deleted the test/writer-backpressure-tdd branch October 10, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant