Skip to content

feat: narrow probe selection by intent: in run.spec - #2113

Open
patriciapampanelli wants to merge 5 commits into
NVIDIA:mainfrom
patriciapampanelli:feature/intent-probe-selection
Open

feat: narrow probe selection by intent: in run.spec#2113
patriciapampanelli wants to merge 5 commits into
NVIDIA:mainfrom
patriciapampanelli:feature/intent-probe-selection

Conversation

@patriciapampanelli

@patriciapampanelli patriciapampanelli commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

intent: in run.spec was accepted but had no effect on probe selection. This PR makes it filter: intent:/-intent: now narrow the candidate set by intent typology descendancy.

Design decisions

  • Matching is by typology descendancy, not string prefix — a leaf code matches only itself, a branch code matches every leaf under it.
  • intent: (include) and -intent: (exclude) filter symmetrically.
  • IntentProbe subclasses (no fixed intent) are pruned only if every included code is explicitly blocked for that probe; otherwise they're kept, since which intents they actually serve is decided later, at generation time.
  • A malformed intent: code is rejected and reported (e.g. on --list_probes/--list_buffs), not silently dropped.
  • Error/warning messages now name the actual intent code(s) involved instead of a generic message.

Verification

  1. garak --list_probes --spec probes.lmrc,intent:S005hate — only lmrc.SlurUsage was listed (leaf code, sibling S005bully excluded).
  2. garak --list_probes --spec probes.dan,-intent:T009 — printed No probes match the provided filter (all active dan.* declare T009ignore).
  3. garak --target_type test.Blank --spec probes.grandma.GrandmaIntent,intent:S005bully — ran to completion with no warning, produced evaluated results.
  4. garak --target_type test.Blank --spec probes.dan.AutoDANCached,intent:T009ignore — no warning
  5. garak --list_probes --spec probes.dan,intent:zzz — printed a "skipped selector" notice for intent:zzz and still listed the full dan.* family.
  6. garak --target_type test.Blank --spec probes.lmrc,intent:M010degrade — aborted with a message naming M010degrade as the code nothing matched.
  7. garak --list_probes --spec probes.lmrc,intent:S005hate --probe_tags avid-effect:performance:P0401 — only lmrc.SlurUsage was listed; the deprecated tag was ignored.
  8. No probes.* at all (default candidate = the whole active registry):
    • garak --list_probes --spec intent:S005hate — 3 probes across 3 different modules (continuation, grandma, lmrc), not one family.
    • garak --list_probes --spec=-intent:T009 — every active probe declaring T009ignore (across dan, goodside, and others) dropped out.
    • garak --list_probes --spec intent:S005,-intent:S005hate,tier:1 — mixed include+exclude plus a tier: filter narrowed to a single probe.

Closes #1873

Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

intent-based probe selection in run.spec

1 participant