fix(analyze): draw bootstrap samples from a local RNG - #2063
Open
chiruu12 wants to merge 1 commit into
Open
Conversation
chiruu12
force-pushed
the
fix/bootstrap-local-rng
branch
from
August 14, 2026 19:47
cd4c2a1 to
36680ce
Compare
Signed-off-by: chiruu12 <103719146+chiruu12@users.noreply.github.com>
chiruu12
force-pushed
the
fix/bootstrap-local-rng
branch
from
August 21, 2026 02:10
36680ce to
90bcb53
Compare
Author
|
The red here is not this diff. The only failure is
The bootstrap change itself is 5691 passed on this run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2062
_bootstrap_calculationseeded the global numpy generator with the run seed. The evaluator calls it once per probe per detector andconfidence_interval_methoddefaults tobootstrap, so on a seeded run the global RNG was reset repeatedly during evaluation and every numpy draw afterwards restarted from the same state.That reaches two consumers: branch shuffling in
red_team/conversation.py, which permutes equal-scoring elements before pruning, and parent selection inautodan/genetic.py. Both stop varying, so--seedwas quietly removing the randomness TAP and AutoDAN use to explore.This draws from
np.random.default_rng(seed)instead. Same seed still gives the same interval, and the global generator is left alone.Added a test that draws from
np.randomaround a CI calculation and compares against the same draws without one. It fails on main and passes here. The existing 14 tests in the file still pass, includingtest_calculate_bootstrap_ci_reproducibility.One thing to note:
default_rnguses PCG64 wherenp.random.seeduses the legacy MT19937, so a seeded run will report slightly different interval bounds than before. Same seed still gives the same answer, it is just not bit identical to previous releases. Happy to usenp.random.RandomState(seed)instead if you would rather keep the old numbers.pytest tests/analyze/is green, 159 passed.