Skip to content

fix(harness): evict idle turn gates and make the lease idempotent - #3533

Open
wahllllll wants to merge 4 commits into
agentscope-ai:mainfrom
wahllllll:fix/session-turn-gate-lease
Open

wahllllll wants to merge 4 commits into
agentscope-ai:mainfrom
wahllllll:fix/session-turn-gate-lease

Conversation

@wahllllll

Copy link
Copy Markdown
Contributor

AgentScope-Java Version

2.1.0-SNAPSHOT

Description

Fixes #3532

Two defects live in the lease that LocalSessionTurnGate.acquire(String) returns. The issue reports the first; the second is in the same three lines and is arguably worse, because it breaks the gate's own contract.

1. Idle keys are never evicted. acquire inserted a Semaphore with computeIfAbsent and the lease only called release(), so every key ever seen stayed in the map and its semaphore could never be collected.

2. close() is not idempotent, and TurnLease says it must be. TurnLease carries "Implementations must be idempotent: calling close() more than once must be safe", but the lease was a bare semaphore::release. Every repeated close leaves one more permit behind, so the next two acquirers for that key can hold it at the same time, and isRunning(key) then reports false while a turn is in flight.

That is not only a theoretical contract. HarnessGateway.withGatedTurn builds its AtomicReference<TurnLease> outside the Mono.defer(...), so a second subscription of the returned Mono runs doFinally twice against the same lease object.

Why this does not use the eviction the issue proposes

The issue suggests releasing, then removing when semaphore.getQueueLength() == 0. That check cannot see a thread that took a just-released permit without ever queueing — Semaphore.acquire() takes an available permit without entering the queue. So:

  1. holder A closes: release(), queue is empty;
  2. thread C barges in and takes that permit (still no queue);
  3. A reads getQueueLength() == 0 and removes the entry while C is inside;
  4. thread D calls acquire, gets a fresh semaphore, and holds the key together with C.

This change instead counts the leases (holders plus waiters) per key and drops the entry when the count reaches zero. The decrement and the removal run inside ConcurrentHashMap.compute, so they are atomic with respect to a concurrent acquire on the same key: an acquirer either joins the existing gate or, once it is gone, builds a fresh one. It can never obtain a second semaphore for a key that still has a holder.

Red / green

Red test on ec9446023 (current main):

[ERROR] LocalSessionTurnGateTest.evictsIdleKeysSoTheMapDoesNotGrowWithoutBound:150 closed leases must not leave stale gate entries behind: 50 ==> expected: <0> but was: <50>
[ERROR] LocalSessionTurnGateTest.closeIsIdempotentSoARepeatedCloseCannotOpenTheGateTwice:169 the second lease must be the only holder ==> expected: <true> but was: <false>
[ERROR] Tests run: 6, Failures: 2, Errors: 0, Skipped: 0

With the change:

[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0 -- in io.agentscope.harness.agent.gateway.LocalSessionTurnGateTest

Validation on commit 725461e:

  • LocalSessionTurnGateTest: 6 passed (the 3 existing tests plus 3 new ones: the map is empty after 50 acquire/close cycles; a repeated close() cannot open the gate twice; and 8 threads × 200 rounds on one key never overlap two holders — that last one passes before and after, and is there to catch an eviction that races a live holder).
  • WakeupDispatcherTest: 5 passed, and ControlPlaneSessionTurnGateTest: 3 passed — the control-plane gate falls back to LocalSessionTurnGate.
  • mvn spotless:apply clean; JDK 17.
  • Not covered: an end-to-end gateway run. The new tests exercise the gate directly.

Checklist

  • Code has been formatted with mvn spotless:apply
  • All tests are passing (mvn test) -- the gate, wakeup and control-plane gate tests passed; the full reactor does not finish in this sandbox
  • Javadoc comments are complete and follow project conventions
  • Related documentation has been updated -- no documentation describes the lease lifecycle
  • Code is ready for review

LocalSessionTurnGate kept one Semaphore per key for the lifetime of the
process: acquire() inserted the entry with computeIfAbsent and the lease
returned by it only called release(), so every key ever seen stayed in the
map and its Semaphore could never be collected.

TurnLease also documents that close() must be idempotent, but the lease
was a bare semaphore::release. Each repeated close left one more permit
behind, so the next two acquirers for that key could hold it at the same
time -- and isRunning() then reported false while a turn was in flight.

Count the leases (holders plus waiters) per key and drop the entry once
that count reaches zero. The decrement and the removal run inside
ConcurrentHashMap.compute, so they are atomic against a concurrent
acquire() on the same key: an acquirer either joins the existing gate or,
once it is gone, builds a fresh one. Eviction is driven by that count
rather than by Semaphore.getQueueLength(), which cannot see a thread that
took a just-released permit without ever entering the queue.
@codecov

codecov Bot commented Oct 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.92308% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...pe/harness/agent/gateway/LocalSessionTurnGate.java 76.92% 4 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@guslegend0510 guslegend0510 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CryoThrust CryoThrust left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked 09582f5e37b643149fb1f29e579e3ec0a7ba1013 locally. Added two interruption controls: a pre-interrupted acquisition leaves no idle entry, and interrupting one of two queued waiters preserves the active holder and surviving waiter, then retires the entry when everyone finishes. The latter waits for both threads to enter the semaphore queue before interrupting one.

All eight LocalSessionTurnGateTest cases pass. The selected Harness/Control Plane dependency reactor passes clean verify, including formatting: 4,396 tests, zero failures/errors, 27 skips across seven modules. This is not the full repository or a live control-plane deployment. Removing only retire(key, gate) from the interruption catch makes both new controls fail; production is restored.

Test-only commit for adoption: CryoThrust@6224c21 . No production changes or competing fix PR.

The current Linux job's reported error is AgentProtocolTaskStoreHitlTest.submit_pausesOnPermissionAsking_andResumeDenyCompletes (task is already running: hitl-1); the six original gate cases pass in that same job. Windows passes. That locates the failure but does not establish its cause or make the full CI green.

@wahllllll

Copy link
Copy Markdown
Contributor Author

Thanks for the independent verification — the eight gate cases and the two interruption controls line up with what I see locally.

On the Linux failure you located (AgentProtocolTaskStoreHitlTest.submit_pausesOnPermissionAsking_andResumeDenyCompletes): I reproduced it deterministically and traced it to a race in AgentProtocolTaskStore.resume(), not to anything in this PR. The window sits between markAwaitingConfirm persisting awaiting_confirm and the run future completing, so a client that reacts to the new status can be rejected with a spurious task is already running. Root cause, a deterministic reproducer, and a suggested two-line fix are in #3552.

Nothing in LocalSessionTurnGate is involved, so this PR's Linux red is unrelated to the diff. I am not opening a fix PR for it myself because AgentProtocolTaskStore.java is already touched by two open PRs (#3013, #2177) — reported instead.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: LocalSessionTurnGate.gates grows without bound — Semaphore entries for completed sessions are never evicted

4 participants