feat(validator): make OSS credibility volume-aware and gate-only#1359
Conversation
|
I appreciate the work on this. Early in my Gittensor run, the maintainer of the main project I contribute to did a major rebase / fork-network switch. That left 4 of my open PRs effectively unrecoverable, so they had to be closed despite the work still being valid. It was pretty painful to get hit that hard on credibility while also having substantially more merged PRs overall. This change feels more balanced. It still filters for contributors with a real track record, but it’s a lot more forgiving for people who are consistently shipping and occasionally get caught by repo/process churn outside their control. Cheers 🔥 |
|
Mostly what I like from the issue is this:
can you scope this down to JUST that? e.g. remove credibility as the per-PR multiplier, I'm not fond of volume base credibility, it again almost tries to promote inundating maintainers by trying to just open more PRs in hopes of raising your weighted credibility tl;dr just remove credibility as a per PR multiplier, have it just gate eligibility |
anderdc
left a comment
anderdc
left a comment
There was a problem hiding this comment.
Per the maintainer's comment, scope this to gate-only. Keep: the credibility_multiplier = 1.0 change in scoring.py. Drop: the volume-aware smoothing — revert calculate_credibility in credibility.py to the flat merged/(merged+closed) ratio, remove CREDIBILITY_SMOOTHING_PRIOR from constants.py, drop the MIN_CREDIBILITY 0.80->0.70 retune, and remove the volume-aware tests. Also rebase onto test to clear the conflicts.
…ius#1340) Past the credibility gate, credibility no longer multiplies each merged PR's earned_score. Eligibility still uses the flat merged/(merged+closed) ratio against the configured min_credibility threshold; nothing else changes. Addresses anderdc's review on entrius#1359: scope reduced to the gate-only piece. The volume-aware smoothing prior, MIN_CREDIBILITY 0.80->0.70 retune, and volume-aware tests are dropped per the requested scope-down. Updates test_per_repo_credibility_multiplier (renamed to test_credibility_is_gate_only_not_a_multiplier) to assert the new gate-only semantics: credibility ratio is still computed, but credibility_multiplier is hardcoded to 1.0 and earned_score is no longer taxed.
plind-junior
force-pushed
the
fix/1340-credibility-smoothing
branch
from
659f4ce to
b21e2f7
Compare
|
@anderdc Fixed the comment |
anderdc
left a comment
anderdc
left a comment
There was a problem hiding this comment.
Don't neutralize the multiplier to 1.0 — remove it entirely. A constant-1.0 field is dead weight that still has to be torn out, so do the full gittensor-side removal while credibility's per-PR role is in scope. Keep credibility itself — calculate_credibility, the eligibility gate, and repo_eval.credibility reporting all stay. Only the per-PR multiplier goes.
Remove:
- oss_contributions/scoring.py — the
pr.credibility_multiplier = 1.0line in_score_eligible_repo_prs. - mirror/scored_pr.py:32 and :83 — the
credibility_multiplierfield and its'cred'entry incalculate_final_earned_score. - mirror/scoring.py:397 — the
scored.credibility_multiplier = 1.0placeholder. - mirror/adapters.py:123 — the
credibility_multiplier=scored.credibility_multipliercopy into the PullRequest DTO. - classes.py:172 — the
credibility_multiplierfield on PullRequest. Itscalculate_final_earned_score(classes.py:200-212) is already dead — no caller — so drop that whole method too. - storage/queries.py:50,78 and storage/repository.py:184 — drop
credibility_multiplierfrom the PR insert/upsert and its bound value.
Update the tests that assert on the multiplier (test_repo_config_fields.py, test_scoring.py, test_adapters.py, test_scored_pr.py).
Leave the gittensor-db column and das/UI fields as they are — those get retired separately.
…#1359 review) Per anderdc's CHANGES_REQUESTED review on entrius#1359: don't neutralize the multiplier to 1.0 — remove the field, its placeholder writes, the DTO copy, the storage binding, and the dead PullRequest.calculate_final_earned_score method. Credibility itself stays exactly as before: calculate_credibility, the eligibility gate, and repo_eval.credibility reporting are unchanged. Removed: - mirror/scored_pr.py: credibility_multiplier field + 'cred' entry - mirror/scoring.py: non-MERGED placeholder write - mirror/adapters.py: DTO copy into PullRequest - oss_contributions/scoring.py: per-PR write in _score_eligible_repo_prs - classes.py: PullRequest.credibility_multiplier field + dead calculate_final_earned_score method (no callers) - storage/queries.py: column from INSERT + UPDATE; storage/repository.py: bound value Tests updated: test_repo_config_fields, test_scoring, test_adapters, test_scored_pr. gittensor-db column and das/UI fields untouched per the review — those are retired separately.
|
Done @anderdc |
3 participants
Closes #1340. Credibility (merged/(merged+closed)) did two jobs — a 0.80 hard gate and a per-PR multiplier — and both punished active contributors. A miner at 30 merged / 8 closed scored 0.79 and was rejected outright, while 3/0 earned a free 1.0; and anyone past the gate was still taxed by the same ratio on every PR (charged twice).
Smoothing prior: credibility = (merged + k) / (merged + closed + 2k), k=CREDIBILITY_SMOOTHING_PRIOR=2. 3/0 -> 0.71 (no free pass), 30/8 -> 0.76 (no death sentence). MIN_CREDIBILITY retuned 0.80 -> 0.70 to match the new scale (still per-repo overridable).
Gate-only: _score_eligible_repo_prs no longer multiplies earned score by credibility (credibility_multiplier=1.0). Past the gate, PRs stand on review-quality, label, and time-decay multipliers. repo_eval.credibility is still computed and reported.
Scope: OSS-contribution credibility only, matching the issue's cited code. The issue-discovery analogue (calculate_issue_credibility + discovery_credibility_multiplier) has the same shape and is left as a follow-up.