fix(telemetry): report each terminal outcome once and stamp usage_context on every event

[santoshkumarradha]

Summary

Investigating an execution_completed spike in product metrics (~7k/day baseline → 70k on Aug 22 → 363k on a partial Aug 23) surfaced two defects in the OSS telemetry client that let the event count things that were not executions. This fixes both: a terminal outcome is now reported once per execution, and usage_context is stamped on every event so CI traffic can be told apart from real usage.

Neither defect is the whole story of that spike — the count is aggregated by a hosted relay outside this repo — but both are real, both inflate the same metric, and both were live during the window.

Duplicate lifecycle events were forwarded verbatim. A terminal status callback re-delivered after a lost 200 used to re-run every side effect in handleStatusUpdate, publishing a second execution_completed for an execution that had already been reported. #951 closed that path in the handler on Aug 23 and measured it directly ("two duplicate succeeded callbacks and one failed callback emitted three execution_completed events"); all three SDKs retry a status callback up to five times, so a single execution could report several. The telemetry client had no defense of its own — it minted a stable telemetry_event_id and delegated deduplication entirely to the ingest side, so any republish reached the wire and counted if ingest did not honor the ID.

handleExecutionEvent now remembers the terminal outcomes it has reported and drops a repeat. Only stable identities are eligible; the random ones assigned in the else branch belong to transitions that are allowed to recur (timeout → running → timeout) and collapsing those would lose real events. The set is bounded at 8192 keys with oldest-first eviction — a duplicate arrives within seconds of the original, so eviction can only ever drop a key long past the window in which it could still suppress anything. Keys are built from raw execution IDs and never leave the process; only the existing HMAC is sent.

usage_context rode only on control_plane_started. A CI job starts the control plane on a fresh volume, so it mints a new install ID and its executions are indistinguishable from a real first-time user's. With the context carried by the startup event alone, no downstream query could separate the two after ingestion — which is why the polluted range cannot be cleaned by filtering and has to be trimmed by date instead. #942 disabled telemetry for the functional-test compose stacks, which fixes one known producer; stamping the property on every event makes every producer distinguishable at the source, including ones we have not found yet. detectUsageContext already existed and already detected CI — it was simply never applied beyond startup.

telemetry_schema_version goes to 2 → 3 so the ingest side can tell a build that stamps usage_context everywhere from one that does not, and therefore know for which date ranges a usage_context filter is trustworthy. This needs confirmation against the relay at agentfield.ai/api/oss/telemetry before merge — if that endpoint validates the version strictly rather than treating it as advisory, a bump would drop events, which is the opposite of what this PR is for.

Type of change

• Bug fix

Test plan

• cd control-plane && go test ./internal/observability/ -count=1
• cd control-plane && go test ./internal/observability/ -count=1 -race (the suppression set is mutex-guarded)
• cd control-plane && go test ./internal/observability/... ./internal/handlers/... ./internal/server/... ./internal/events/... -count=1 — all green
• cd control-plane && go build ./... && go vet ./internal/observability/

New tests, all in telemetry_test.go:

• TestTelemetryTerminalOutcomeReportedOnce — four identical completed/failed/cancelled events enqueue one.
• TestTelemetryTerminalSuppressionIsPerExecution — two executions completing both report.
• TestTelemetryNonTerminalEventsAreNotSuppressed — a repeated execution_started is not collapsed.
• TestTelemetryReportedSetEvictsOldestPastCapacity — bounded growth, oldest-first eviction.
• TestTelemetryUsageContextStampedOnEveryEvent — execution, node, and plain enqueued events all carry it.
• TestTelemetryDetectsUsageContext — each CI variable, plus server and dev_or_local. Clears the ambient CI environment first, or every case would pass on the runner's own CI=true.

TestTelemetryExecutionEventIdentityIsStableAndOpaque changed rather than being deleted: it used to send the same event twice and read two queue entries, which now blocks because the duplicate is suppressed. It recomputes the identity a republish would carry and asserts it matches the one already sent — the property that actually mattered (ingest can recognize a re-delivery as the same outcome) — and keeps the payload-opacity assertions unchanged.

Existing coverage that pins behavior this PR deliberately does not change: TestTelemetryTimeoutEventIdentityDoesNotCollapseRepeatedTransitions and TestTelemetryExecutionEventIdentityDoesNotCollapseMissingIDs both still pass.

Test coverage

• I ran tests for the surface(s) I changed locally.
• New code paths are covered by tests in this PR (no bare additions).
• No code was removed, so coverage-baseline.json is untouched.
• The coverage gate check is green in CI before requesting review.

Notes for review

• Not changed deliberately: telemetry still defaults to enabled in CI. Stamping usage_context keeps the data and makes it filterable, which seemed strictly better than dropping it; turning collection off by default in CI is a product call, and the opt-out is now documented instead.
• The suppression is in-process only. A control plane restart forgets what it reported, which is correct — the stable telemetry_event_id remains the cross-process idempotency mechanism, and this is a second line of defense, not a replacement.
• telemetryReportedSet's zero value is usable and allocates on first observe, so a caller constructing TelemetryService directly (as every test in the package does) gets the same behavior as production rather than silently getting none.

Related issues / PRs

• Follows #951 (made terminal status callbacks idempotent — fixed the producer; this hardens the consumer).
• Follows #942 (stopped functional-test runs reporting to production telemetry — fixed one CI producer; this makes all of them identifiable).

  

[github-actions]

📊 Coverage gate

Thresholds from .coverage-gate.toml: per-surface ≥ 84%, aggregate ≥ 85%, max per-surface regression ≤ 1.0 pp, max aggregate regression ≤ 0.50 pp.

Surface	Current	Baseline	Δ
control-plane	87.20%	87.40%	↓ -0.20 pp	🟡
sdk-go	92.90%	92.00%	↑ +0.90 pp	🟢
sdk-python	94.20%	93.73%	↑ +0.47 pp	🟢
sdk-typescript	91.39%	90.42%	↑ +0.97 pp	🟢
web-ui	84.76%	84.79%	↓ -0.03 pp	🟡
aggregate	85.66%	85.75%	↓ -0.09 pp	🟡

✅ Gate passed

No surface regressed past the allowed threshold and the aggregate stayed above the floor.

[github-actions]

📐 Patch coverage gate

Threshold: 80% on lines this PR touches vs origin/main (from .coverage-gate.toml:thresholds.min_patch).

Surface	Touched lines	Patch coverage	Status
control-plane	65	95.00%	✅
sdk-go	0	—	➖ no changes
sdk-python	0	—	➖ no changes
sdk-typescript	0	—	➖ no changes
web-ui	0	—	➖ no changes

✅ Patch gate passed

Every surface whose lines were touched by this PR has patch coverage at or above the threshold.