Skip to content

Fence off the play_events rows that mean 0.5 by construction - #80

Merged
jeffcrouse merged 1 commit into
mainfrom
docs/feedback-data-cutoff
Aug 3, 2026
Merged

Fence off the play_events rows that mean 0.5 by construction#80
jeffcrouse merged 1 commit into
mainfrom
docs/feedback-data-cutoff

Conversation

@jeffcrouse

Copy link
Copy Markdown
Member

Phase 1. Three decisions wait on ADR-0004's data — its own completion-ratio threshold, ADR-0005's RADIO weights, and #53 — and the data doesn't start where we assumed.

What the live database says

Until #57, the web client delivered a play the moment listening crossed min(duration / 2, 4 min) and sent completion_ratio as measured at that instant, never revising it. Verified again on 2026-08-02, by day and context:

completions avg
every context before 2026-08-01 (incl. library) 0.42 – 0.50
library, 2026-08-01 → 02 0.972 / 1.000 (skips ≤ 0.1)

795 of 823 rows predate the fix. library is contaminated too because the web derives context from the queue source and sends library for a library queue — exactly as the native app does.

So the clock restarted on 2026-08-01, not 2026-07-27. A month of trustworthy data lands around 2026-09-01, and all three decisions inherit that date.

Why a date, and why not delete

FEEDBACK_TRUSTWORTHY_SINCE / trustworthy_feedback_only put the boundary one call away instead of leaving it to discipline.

A date, not a client filter: play_events records no client and context doesn't stand in for one, so the good native rows can't be separated from the bad web rows beside them. Selecting on the ratio would be circular — it's the variable being measured. The cutoff costs those native rows; nothing cheaper does.

Excluded, not deleted, for a reason the investigation turned up: the damage isn't limited to the ratio. A track abandoned at 55% was recorded as a completion at ~0.5 rather than as a skip, so outcome is unreliable before the cutoff and skips are under-counted. What survives is that skipped/rejected rows describe real abandonments — an incomplete census, not a wrong one.

The reassuring part

ambient._negative_signal counts exactly skipped and rejected, over a rolling 90-day window. The live recommender was never poisoned — it's been running on a slightly weak negative signal, which heals as the window passes the cutoff (~2026-10-30). Nothing reads completion_ratio anywhere yet, so the threshold work is entirely ahead of us.

Also

  • ADR-0005 gains the Implementation: block it never had — which is why it looked unbuilt in a survey while RADIO had been shipping for weeks.
  • Volume: 28 events / 26 distinct tracks over two clean days. A month ≈ 400 events — enough to place a threshold, thin for weight tuning.
  • Two follow-ups recorded: a nullable client column (cheap now, worthless retroactively — the one thing that would have made this separable), and that whether the web still reports at all is unverified, since every post-cutoff row carries the context the native app hardcodes. Settle that by playing one track in the web app, not by reasoning about rows.

Verification

Seven unit tests on the boundary, including that it's inclusive and naive-UTC to match TIMESTAMP WITHOUT TIME ZONE. ruff clean. The DB-backed suite can't run here — no local postgres on 5432 — so CI covers it; the new module is pure and needs none.

🤖 Generated with Claude Code

https://claude.ai/code/session_014P9p2fvFnyiywBxGkv4gfW

…ruction

Phase 1. Three decisions wait on ADR-0004's data — its own completion-ratio
threshold, ADR-0005's RADIO weights, and #53 — and the data does not start where
everyone assumed.

Until #57 the web client delivered a play the moment listening crossed
`min(duration / 2, 4 min)` and sent `completion_ratio` as measured at that
instant, never revising it. Verified again on the live database on 2026-08-02, by
day and context: **795 of 823 rows predate the fix**, and every context before
2026-08-01 shows completions averaging 0.42–0.50 — including `library`, because
the web derives context from the queue source and sends `library` for a library
queue exactly as the native app does. From 2026-08-01 the same context reads
0.972 and 1.000, with skips at ≤0.1.

So the clock restarted on 2026-08-01, not 2026-07-27: a month of trustworthy data
lands around 2026-09-01.

`FEEDBACK_TRUSTWORTHY_SINCE` and `trustworthy_feedback_only` make the boundary one
call away instead of a matter of discipline. A date, not a client filter, because
`play_events` records no client and `context` does not stand in for one — the good
native rows cannot be separated from the bad web rows beside them, and selecting
on the ratio would be circular.

Excluded rather than deleted, for a reason the investigation turned up: the damage
is not limited to the ratio. A track abandoned at 55% was recorded as a completion
at ~0.5 rather than as a skip, so `outcome` is unreliable before the cutoff and
skips are *under*-counted. What survives is that `skipped` and `rejected` rows
describe real abandonments. `ambient._negative_signal` counts exactly those two
over a rolling 90-day window — so **the live recommender was never poisoned**, it
has been running on a slightly weak negative signal that heals around 2026-10-30.

ADR-0005 also gains the `Implementation:` block it never had, which is why it
looked unbuilt in a survey while `RADIO` had been shipping for weeks.

Two follow-ups recorded: a `client` column, cheap now and worthless retroactively,
which is the only thing that would have made this separable; and that whether the
web still reports at all is unverified — every post-cutoff row carries the context
the native app hardcodes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014P9p2fvFnyiywBxGkv4gfW
@jeffcrouse
jeffcrouse merged commit 125752f into main Aug 3, 2026
12 of 13 checks passed
@jeffcrouse
jeffcrouse deleted the docs/feedback-data-cutoff branch August 3, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant