feat(evaluators): add pass@k attack success rate metric

[stefanoamorelli]

Adds pass@k for attack success rate (implements #1911).

Example, 5 prompts, each breached on 1 of 5 generations:

attack success rate:  20.00%  pass@1: 20.00%  pass@5: 100.00%

A new pass_at_k field is written alongside it in the eval report entry and printed on the CLI.

This setting can be configured via reporting.pass_at_k (default [1, 5], an empty list [ie. []] disables it).

This is an additive change and the existing pooled ASR remains unchanged.

Config examples

Default [1, 5], on a run with run.generations: 5 (5 prompts, each breached on 1 of 5 generations):

(attack success rate:  20.00%  pass@1: 20.00%  pass@5: 100.00%)

"pass_at_k": {"1": {"score": 0.2, "prompts": 5}, "5": {"score": 1.0, "prompts": 5}}

More k values, on a run with run.generations: 10 (5 prompts, each breached on 2 of 10 generations):

reporting:
  pass_at_k: [1, 5, 10]

(attack success rate:  20.00%  pass@1: 20.00%  pass@5: 77.78%  pass@10: 100.00%)

"pass_at_k": {"1": {"score": 0.2, "prompts": 5}, "5": {"score": 0.7778, "prompts": 5}, "10": {"score": 1.0, "prompts": 5}}

Disabled: the CLI suffix is dropped and no pass_at_k field is written.

reporting:
  pass_at_k: []

(attack success rate:  20.00%)

Tests

$ python -m pytest tests/analyze/test_pass_at_k.py tests/evaluators/test_evaluators.py
77 passed in 0.39s

$ python -m pytest tests/test_config.py tests/test_docs.py
765 passed in 4.81s

---

Draft for review. I used claude code for this. I reviewed every line and take responsibility for the change.

[jmartin-tech]

Awesome contribution, the maintainer team took a bit of time working thru what this metric means and this scoring pattern seems like it would surface a better understanding of the run results.

One item I see missing is that this only incorporates these values for terminal output. The most common artifact reviewed is understood to be the report.html. This PR's scope is not required to expand to expose this in that space however it should likely expand to define how and where in the final digest object written to report.jsonl this might be able to provide the same values being presented on stdout.

Further confirmation of how the scores function is stil ongoing but the overall feedback should enable iteration while that review is completed.

[jmartin-tech]

Given that terminology in garak, while I understand this diverges from the paper the metric here is probably more clear as hit@k for the following reasons:

• detector.always.Pass refers maps to scoring as 0.0
• the detector base class refers to a successful result as a hit and a pass has opposite semantics
• Detections for closer evaluation are logged in hitlog.jsonl further aligning this product's terminology.

[jmartin-tech]

As noted in another comment garak's jargon would align better as hit_at_k.

These seem like reasonable values, and the 5 here matches to the default generations value. Do you think there is value in having this configuration be for expanded coverage instead of all values to be used?

Consider that a default behavior where the number of generations the run was launched for would always include the hit@k where k == generations or more simply the for an Attempt we could always report at least k = len(self.outputs). Then augment that set with any additional values in reporting.hit_at_k that are less than the number outputs in the Attempt.

I could see some probes this might not align to generations such as atkgen.Tox though if the upper bound on k is based on len(Attempt.outputs) that might turn out to be a non-issue.

With the expectation that this is the number of inference generations aligned to at least one hit identified this metric seem very helpful.