Presentation polish: CI, unit tests, CONTRIBUTING, submission path - #24
Conversation
Adds the table-stakes a public LLM benchmark is expected to have, on-brand with the simple-README style: - .github/workflows/ci.yml — CI on Python 3.9/3.11/3.12 running the offline smoke test, the CTF demo, and pytest. Gives a green "tests passing" badge and guards against regressions. - tests/test_ctf.py — deterministic offline unit tests for Family N: outcome classification, None-not-NaN strict-JSON scoreboard, rate ranges, BlueMonitor hit-consistency, the first-clean-query threshold, control/primary tiers. - CONTRIBUTING.md — the submission path (add your NLA -> run -> attribute -> PR a result that clears a null control) plus dev setup and house rules. - README — CI + python badges, an "Add your NLA" leaderboard-submission pointer, and a CONTRIBUTING entry in the docs table. No library behavior change. pytest green locally (7 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMuBSoDsAPwNSvev9nmvVr
There was a problem hiding this comment.
Code Review
This pull request adds a comprehensive contributing guide (CONTRIBUTING.md), updates the main README.md with CI/Python badges and links to the contribution guidelines, and introduces a new unit test suite (tests/test_ctf.py) for the CTF Red/Blue family (Family N). The reviewer suggested enhancing the oracle tier test by adding functional assertions to verify the behavior of the returned oracle callables rather than just checking their tier strings.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| def test_probe_oracle_is_primary_tier(): | ||
| _oracle, tier = ctf.probe_capture_oracle(lambda _t: 0.9) | ||
| assert tier == "primary" | ||
| _oracle2, tier2 = ctf.survival_oracle() | ||
| assert tier2 == "control" |
There was a problem hiding this comment.
The test test_probe_oracle_is_primary_tier only asserts the tier string returned by probe_capture_oracle and survival_oracle, but does not verify that the returned CaptureOracle callables actually function and evaluate inputs correctly. Adding assertions to call these oracles and verify their boolean return values would prevent silent regressions in the oracle logic.
| def test_probe_oracle_is_primary_tier(): | |
| _oracle, tier = ctf.probe_capture_oracle(lambda _t: 0.9) | |
| assert tier == "primary" | |
| _oracle2, tier2 = ctf.survival_oracle() | |
| assert tier2 == "control" | |
| def test_probe_oracle_is_primary_tier(): | |
| _oracle, tier = ctf.probe_capture_oracle(lambda _t: 0.9) | |
| assert tier == "primary" | |
| assert _oracle("anything", None) is True | |
| _oracle_low, _ = ctf.probe_capture_oracle(lambda _t: 0.1) | |
| assert _oracle_low("anything", None) is False | |
| _oracle2, tier2 = ctf.survival_oracle() | |
| assert tier2 == "control" | |
| flag = ctf.CTF_FLAGS[0] | |
| assert _oracle2(flag.objective, flag) is True | |
| assert _oracle2("completely unrelated text", flag) is False |
Adds the table-stakes a public LLM benchmark is expected to have, kept on-brand with the simple-README style. No library behavior change.
What's added
.github/workflows/ci.yml) — runs the offline smoke test, the CTF demo, andpyteston Python 3.9 / 3.11 / 3.12. Gives a green "tests passing" badge and guards against regressions (the repo had no CI before).tests/test_ctf.py) — deterministic, offline tests for Family N: outcome classification, the None-not-NaN strict-JSON scoreboard, rate ranges,BlueMonitorhit-consistency, the first-clean-query threshold, and control/primary tiers.7 passedlocally.Why
The three biggest gaps vs. MMLU/HELM/SWE-bench-style projects were: no CI/tests, no contributor/submission path, and a leaderboard not framed as one. This closes them without bloating the README.
🤖 Generated with Claude Code
Generated by Claude Code