fix(schedular): a dead calendar workbench resets instead of failing forever

[hugolytics]

The incident

google-calendar-mcp crash-looped from 09:13 today on a zero-byte tokens.json
(JSON.parse("") → "Unexpected end of JSON input"). The Slack bot started at
11:24 while it was still down.

Every Add to calendar press then failed with MCP Actor not running, call initialize() first — and so did Try again, repeatedly. Restoring the token
file and restarting the container was not enough: the bot stayed broken until it
was restarted too.

Why a transient outage became a permanent failure

PlannerAgent._ensure_workbench cached one McpWorkbench for the process
lifetime, with no health check and no reset:

def _ensure_workbench(self) -> McpWorkbench:
    if self._workbench:
        return self._workbench
    ...

AutoGen's McpWorkbench.call_tool lazily calls start() only when
self._actor is falsy. Once an actor exists but its session has died, every
later call raises forever. Nothing in the planner path could recover.

McpCalendarClient had already solved exactly this — recoverable-error markers,
a workbench reset, and a retry-once loop. The planner simply never got the same
treatment. This ports it, and moves the shared half somewhere both can read so
they cannot drift apart again.

The write-safety decision

handle_upsert_calendar_event writes to a real calendar, so blanket retry was
not an option.

call	retries?	why
list-events, get-event	yes	reads; resending is free
delete-event	yes	idempotent
create-event, update-event	no	resets the workbench so the next press works, but never resends

MCP Actor not running is raised locally, before any request leaves the
process, so resending after that marker would in fact be safe. But the same
classifier also matches response timeouts and dropped connections, where the
first request may already have reached Google. One classifier covers all the
markers, so the writes take the conservative branch: a duplicate event on a
real calendar is worse than the bug this fixes. The reset still happens, so a
user's next Try again meets a healthy workbench.

Tests

Six regression tests, each observed failing before the implementation existed:
one reset and one retry on a recoverable error; immediate propagation with no
reset on a non-recoverable one; a retry budget of exactly one; and a test that
fails if anyone later makes creates retry blindly.

584 passed, 6 skipped across the planner/calendar/mcp/schedular slice.

Not included

The container-side cause is already fixed live and is not code: two compose
files create two token volumes (infra_ vs admonish-1_), and the one actually
mounted got zeroed. Worth reconciling separately — the same split can silently
strand credentials again.

🤖 Generated with Claude Code