Skip to content

fix(analyze): draw bootstrap samples from a local RNG - #2063

Open
chiruu12 wants to merge 1 commit into
NVIDIA:mainfrom
chiruu12:fix/bootstrap-local-rng
Open

fix(analyze): draw bootstrap samples from a local RNG#2063
chiruu12 wants to merge 1 commit into
NVIDIA:mainfrom
chiruu12:fix/bootstrap-local-rng

Conversation

@chiruu12

Copy link
Copy Markdown

Fixes #2062

_bootstrap_calculation seeded the global numpy generator with the run seed. The evaluator calls it once per probe per detector and confidence_interval_method defaults to bootstrap, so on a seeded run the global RNG was reset repeatedly during evaluation and every numpy draw afterwards restarted from the same state.

That reaches two consumers: branch shuffling in red_team/conversation.py, which permutes equal-scoring elements before pruning, and parent selection in autodan/genetic.py. Both stop varying, so --seed was quietly removing the randomness TAP and AutoDAN use to explore.

This draws from np.random.default_rng(seed) instead. Same seed still gives the same interval, and the global generator is left alone.

Added a test that draws from np.random around a CI calculation and compares against the same draws without one. It fails on main and passes here. The existing 14 tests in the file still pass, including test_calculate_bootstrap_ci_reproducibility.

One thing to note: default_rng uses PCG64 where np.random.seed uses the legacy MT19937, so a seeded run will report slightly different interval bounds than before. Same seed still gives the same answer, it is just not bit identical to previous releases. Happy to use np.random.RandomState(seed) instead if you would rather keep the old numbers.

pytest tests/analyze/ is green, 159 passed.

Copilot AI lite review requested due to automatic review settings August 14, 2026 18:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Signed-off-by: chiruu12 <103719146+chiruu12@users.noreply.github.com>
@chiruu12
chiruu12 force-pushed the fix/bootstrap-local-rng branch from 36680ce to 90bcb53 Compare August 21, 2026 02:10
@chiruu12

Copy link
Copy Markdown
Author

The red here is not this diff. The only failure is tests/generators/test_anthropic.py::test_anthropic_generator, and the rest of the matrix was cancelled by fail-fast rather than failing on its own.

anthropic 1.0.0 shipped on 20 Aug and moved the SDK onto httpx2, which respx cannot patch, so that test now calls the live API and gets a 401. Nightly on main caught the same thing on an unchanged commit. Filed as #2099 with the cap in #2100.

The bootstrap change itself is 5691 passed on this run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI calculation reseeds the global numpy RNG, so --seed makes later numpy draws repeat

2 participants