Skip to content

Presentation polish: CI, unit tests, CONTRIBUTING, submission path - #24

Merged
SolshineCode merged 1 commit into
masterfrom
claude/v2-presentation-polish
Jul 27, 2026
Merged

Presentation polish: CI, unit tests, CONTRIBUTING, submission path#24
SolshineCode merged 1 commit into
masterfrom
claude/v2-presentation-polish

Conversation

@SolshineCode

Copy link
Copy Markdown
Owner

Adds the table-stakes a public LLM benchmark is expected to have, kept on-brand with the simple-README style. No library behavior change.

What's added

  • CI (.github/workflows/ci.yml) — runs the offline smoke test, the CTF demo, and pytest on Python 3.9 / 3.11 / 3.12. Gives a green "tests passing" badge and guards against regressions (the repo had no CI before).
  • Unit tests (tests/test_ctf.py) — deterministic, offline tests for Family N: outcome classification, the None-not-NaN strict-JSON scoreboard, rate ranges, BlueMonitor hit-consistency, the first-clean-query threshold, and control/primary tiers. 7 passed locally.
  • CONTRIBUTING.md — the submission path a benchmark needs: implement one adapter method → run → attribute to the NLA → open a PR with a result that clears a null control. Plus dev setup and the two house rules (null controls; freeze-on-release).
  • README — CI + Python badges, an "Add your NLA" leaderboard-submission pointer under the results table, and a CONTRIBUTING row in the docs table.

Why

The three biggest gaps vs. MMLU/HELM/SWE-bench-style projects were: no CI/tests, no contributor/submission path, and a leaderboard not framed as one. This closes them without bloating the README.

🤖 Generated with Claude Code


Generated by Claude Code

Adds the table-stakes a public LLM benchmark is expected to have, on-brand
with the simple-README style:

- .github/workflows/ci.yml — CI on Python 3.9/3.11/3.12 running the offline
  smoke test, the CTF demo, and pytest. Gives a green "tests passing" badge and
  guards against regressions.
- tests/test_ctf.py — deterministic offline unit tests for Family N: outcome
  classification, None-not-NaN strict-JSON scoreboard, rate ranges, BlueMonitor
  hit-consistency, the first-clean-query threshold, control/primary tiers.
- CONTRIBUTING.md — the submission path (add your NLA -> run -> attribute ->
  PR a result that clears a null control) plus dev setup and house rules.
- README — CI + python badges, an "Add your NLA" leaderboard-submission pointer,
  and a CONTRIBUTING entry in the docs table.

No library behavior change. pytest green locally (7 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMuBSoDsAPwNSvev9nmvVr

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a comprehensive contributing guide (CONTRIBUTING.md), updates the main README.md with CI/Python badges and links to the contribution guidelines, and introduces a new unit test suite (tests/test_ctf.py) for the CTF Red/Blue family (Family N). The reviewer suggested enhancing the oracle tier test by adding functional assertions to verify the behavior of the returned oracle callables rather than just checking their tier strings.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread tests/test_ctf.py
Comment on lines +76 to +80
def test_probe_oracle_is_primary_tier():
_oracle, tier = ctf.probe_capture_oracle(lambda _t: 0.9)
assert tier == "primary"
_oracle2, tier2 = ctf.survival_oracle()
assert tier2 == "control"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The test test_probe_oracle_is_primary_tier only asserts the tier string returned by probe_capture_oracle and survival_oracle, but does not verify that the returned CaptureOracle callables actually function and evaluate inputs correctly. Adding assertions to call these oracles and verify their boolean return values would prevent silent regressions in the oracle logic.

Suggested change
def test_probe_oracle_is_primary_tier():
_oracle, tier = ctf.probe_capture_oracle(lambda _t: 0.9)
assert tier == "primary"
_oracle2, tier2 = ctf.survival_oracle()
assert tier2 == "control"
def test_probe_oracle_is_primary_tier():
_oracle, tier = ctf.probe_capture_oracle(lambda _t: 0.9)
assert tier == "primary"
assert _oracle("anything", None) is True
_oracle_low, _ = ctf.probe_capture_oracle(lambda _t: 0.1)
assert _oracle_low("anything", None) is False
_oracle2, tier2 = ctf.survival_oracle()
assert tier2 == "control"
flag = ctf.CTF_FLAGS[0]
assert _oracle2(flag.objective, flag) is True
assert _oracle2("completely unrelated text", flag) is False

@SolshineCode
SolshineCode merged commit abcb391 into master Jul 27, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants