experiment: integrate audio S2S probe stack (PRs 1-6) - #7
Closed
codingshreyash wants to merge 145 commits into
Closed
experiment: integrate audio S2S probe stack (PRs 1-6)#7codingshreyash wants to merge 145 commits into
codingshreyash wants to merge 145 commits into
Conversation
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
This allows the user control over what prompt generation parameters maybe overridden for inference calls from the `promptinject` family of probes. Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
Add dedicated test files for donotanswer, grandma, phrasing, realtoxicityprompts, and sata probes. Each covers plugin loading, prompt generation, and module-specific behavior (inheritance, pruning, NLTK masking, dynamic class enumeration). Signed-off-by: boao.dong <markdba313@gmail.com>
Implements AdaptiveAttacks and AdaptiveAttacksFull probes based on
Andriushchenko, Croce, and Flammarion, "Jailbreaking Leading
Safety-Aligned LLMs with Simple Adaptive Attacks" (ICML 2024,
arXiv:2404.02151).
The v1 probe is template-only: it combines a paraphrased adaptive
prompt template (direct-assistant role + structured response scaffold
+ no-"I" affirmative-opener constraint) with a small cached set of
seed forbidden behaviors and pre-computed adversarial suffixes drawn
from public AdvBench results. No gradient computation or online
search is performed.
- garak/probes/adaptiveattacks.py: AdaptiveAttacks (active, capped)
and AdaptiveAttacksFull (uncapped, opt-in)
- garak/data/adaptive_attacks/{seeds,suffixes}.txt: 20 seeds,
3 suffix variants
- tests/probes/test_probes_adaptiveattacks.py: 8 tests covering
load, prompt content, template markers, and prompt-cap behavior
Smoke run against test.Blank completes without error.
Closes NVIDIA#583.
Signed-off-by: Neeraj Kumar Singh <b.neerajkumarsingh@gmail.com>
Address @jmartin-tech review on NVIDIA#1742: (a) Use the canonical follow_prompt_cap guard pattern (per NVIDIA#1546/NVIDIA#1562 + encoding.py / latentinjection.py) instead of an unconditional self._prune_data() call. (b) Add DEFAULT_PARAMS override on AdaptiveAttacksFull to flip follow_prompt_cap to False, replacing the previous Probe.__init__ bypass. This makes the no-cap behavior a configurable param rather than a constructor hack, matches how every other Full variant in the repo expresses uncapped intent, and lets users re-enable the cap on Full via config if they want. AdaptiveAttacks: follow_prompt_cap=True -> prunes to soft_probe_prompt_cap AdaptiveAttacksFull: follow_prompt_cap=False -> keeps every (seed, suffix) pair All 8 tests in tests/probes/test_probes_adaptiveattacks.py still pass, plus tests/test_docs.py adaptive cases (4) and the broader probe suite (1130 passing; the 1 remaining failure is the pre-existing audio probe optional-deps issue unrelated to this PR). Signed-off-by: Neeraj Kumar Singh <b.neerajkumarsingh@gmail.com>
…ests Pare down tests to validation of functionality unique to each probe module, per review feedback. Generic instantiation and prompt-not-empty checks are already covered by plugins/test_plugin_load.py. Signed-off-by: boao.dong <markdba313@gmail.com>
Signed-off-by: boao.dong <markdba313@gmail.com>
Signed-off-by: Jonghyeok Kim <jka236@sfu.ca>
- Dynamically discover evaluator classes by scanning garak.evaluators modules - Replace class-based test grouping with bare test functions - Extract shared constants (THRESHOLD_VALUES, DEFAULT_PROBE, DEFAULT_GOAL, etc.) - Add failure messages to all assert statements - Replace meaningless "G" mock return with realistic emoji symbol - Document fixture parallelisation constraints in docstring - Separate evaluator-specific tests from generic structure tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Ian Miller <milleryan2003@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Ian Miller <milleryan2003@gmail.com>
) Adds a regression guard for the bug fixed in PR NVIDIA#1544 where calling load_base_config() followed by load_config() caused garak.core.yaml to be appended twice to _config.config_files, making it appear twice in HTML reports. The fix (deduplication check in _load_config_files) is already merged, but there was no test to prevent the regression from creeping back in. Closes NVIDIA#1249 Signed-off-by: ppradyoth <pradyoth0@gmail.com>
generators.function.Single overrode DEFAULT_PARAMS with only {"kwargs": {}},
so it did not pick up the base Generator.DEFAULT_PARAMS values. Because
_apply_missing_instance_defaults iterates over the most-derived
DEFAULT_PARAMS, params such as max_tokens, temperature, top_k, context_len,
skip_seq_start and skip_seq_end were never set as instance attributes.
Accessing them (e.g. generator.max_tokens) raised AttributeError.
Merge Generator.DEFAULT_PARAMS into the class dict, matching the pattern
used by the other generators (openai, rest, huggingface, cohere, ollama,
etc.). Multiple inherits from Single and is covered as well.
Adds a regression test asserting both Single and Multiple expose the
inherited defaults while keeping the function-specific kwargs default.
Fixes NVIDIA#1096
Signed-off-by: Aditya Singh <adisin650@gmail.com>
…VIDIA#830) When a user passes a module-name spec (e.g. `-p test`) that maps only to plugins marked `active = False`, `parse_plugin_spec` returns the clause in the rejected list and the CLI raises `Unknown probes: test`. That message is misleading: the module exists, but every plugin in it is inactive. Detect this case at the CLI rejection site by re-enumerating plugins and checking whether any inactive entries share the rejected clause as a namespace prefix. When they do, surface a message that names the module and points the user at calling specific plugins by name. Mixed rejections (some inactive-only, some truly unknown) still report both. The change is contained to `garak/cli.py` so `parse_plugin_spec` keeps its existing `(found, rejected)` signature and `--list_probes` callers are unaffected. A regression test exercises `-p test` and asserts the new wording. Signed-off-by: notnick2 <varun024123@gmail.com>
Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
- NVAudioTranscription now validates inline (raw-bytes) audio against audio_formats using the mime the message carries, and forwards the real content type, instead of labelling every payload as WAV. - OpenAICompatible.audio_formats is derived from audio_mime_subtype_formats' values so supported_formats() cannot advertise a format the request builder is unable to send. Adds tests for both paths. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Support library for audio probes: text-to-speech synthesis over pluggable TTS backends, a deterministic transform-recipe system for perturbing WAV audio with provenance digests, WAV/intelligibility validation, and attack helpers (file metadata, Wilson confidence intervals, per-record summarisation). Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
summarize_audio_groups counted a candidate as a success at score > 0.5 while summarize_audio_records uses >= success_threshold (default 0.5), so the two aggregations disagreed for a score of exactly 0.5. Both now treat 0.5 as a success. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
…aries validate_candidate did a plain substring check, so a required phrase 'ignore' was satisfied by the transcript word 'ignored', letting an invalid candidate pass as scoreable. Phrases now match on whole-word boundaries. Adds a test. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
…lidation An empty required_transcript_phrases list previously short-circuited to scoreable=True, bypassing the WER/semantic checks. It now falls through to normal validation, so an empty list means 'no phrase requirement' rather than 'auto-pass'. Adds a regression test. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Adds NVVoiceChat, a generator for NVIDIA's Nemotron VoiceChat speech-to-speech model via an OpenAI-compatible /v1/chat/completions shim that accepts base64 WAV and returns a text transcript (optionally a WAV reply). The endpoint is set via uri. Appends configurable trailing silence so the model can finish its response before the stream closes, and fails cleanly on non-WAV input. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
_save_response_audio no longer raises when response_audio_dir is set but audio was disabled via extra_body (generate_audio: False) -- it returns cleanly since there is nothing to save. Adds a regression test for the combination, and inlines the format guard in _call_model. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
…s shim Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
…ODEL DEFAULT_MODEL was non-standard (only used by these two nim additions). Follow the OpenAICompatible convention: no hardcoded default, require --target_name and raise a clear error if unset. Avoids shipping a fake or product-specific default. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
…it__ Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Adds PETTS, the base probe that renders an intent as speech and sends it to an audio-capable target to test whether moving an intent from text into spoken audio changes safety behaviour (credited to Rich Harang), plus its tool-risk, reliability, and instruction-priority subclasses. Adds the detectors these score with, including an output-quality screen that discards mis-transcribed audio and a policy-grounded model-as-judge for tool-risk actions. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
…se positive - ToolRiskInstructionInjection now maps the 'suffix' mode to the 'concat' transform, so suffix injection produces results instead of silently raising an unknown-transform error that _audio_prompts swallows. - AudioOutputQualityDetector no longer double-counts mid-word caps as both a fragmentation and a context signal, so coherent answers containing CamelCase product names (macOS, iPhone, JavaScript) are not flagged as word salad. Adds a regression test. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
AudioSuffixInjection appends a hidden spoken instruction to a benign carrier clip the attacker cannot otherwise control; AudioOverlayInjection mixes a hidden instruction into the background of a benign clip. Both score with the tool-risk detectors. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
The suffix/overlay adversarial-condition assertions compared a condition set against its own complement, so they held regardless of the probe's behaviour. They now invoke the prestore hook and assert the is_adversarial flag it stores against a fixed per-condition truth table, catching regressions in the logic. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
PairedDirect varies the signal (speed/noise/filter recipes) and AcousticVoiceBestOfN synthesises the same request across TTS speaker presets. A source counts as compromised if any intelligible candidate succeeds; every candidate passes the independent-ASR intelligibility gate first. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
audio_bon now only sets attempt.notes[attack_goal] on harmful trials, matching audio_acoustic, so benign controls are not mislabelled as adversarial. Adds the missing SPDX header and a non-empty-name guard for candidate_names, plus a regression test. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
PETTS and its subclasses inherit IntentProbe but override _populate_intents, _populate_stubs, and build_prompts with hardcoded data, so the real intentservice is never called. This shim satisfies the import dependency without pulling in the full feature/technique_intent branch.
_config.cas was missing (added by feature/technique_intent); add the GarakSubConfig entry so probe tests that reference _config.cas.intent_spec don't AttributeError. Also flesh out intentservice.load() with a minimal stub set so the base PETTS probe test can exercise the probe() flow.
Implements the full duplex-specific attack surface extension on top of the existing PETTS / NVVoiceChat foundation: EPIC-0 — Foundation (TICKETS 001 + 002) - garak/resources/audio/session.py: SessionScript DSL with SessionEvent, ReactivePattern, and SessionResult; stable SHA-256 digests for provenance - DuplexCapable mixin in nim.py: capability flag + run_session() contract - NVDuplexChat generator: sequential-simulation duplex over any /v1/chat/completions shim; maintains full conversation history across turns; handles reactive triggers (fires interrupt when agent transcript matches a pattern) EPIC-1 — Barge-in as attack primitive (TICKETS 101-103) - PETTSBargeInterrupt: timed refusal-reversal; ablates over all affirmation phrasings per intent; RefusalReversalDetector scores pre/post split - PETTSBargeFlood: rapid N-flood at configurable intervals around the 0.64 s latency window; StateDestabilizationDetector as secondary signal - PETTSFalseBargeIn: injects non-speech false-trigger audio (breath / throat-clear) mid-response via PR-2 transform recipes EPIC-2 — Cross-stream leakage (TICKETS 201-202) - AudioVoiceBleed: watermarks user audio with a pure-tone fingerprint; VoiceBleedDetector scores MFCC cosine similarity shift vs. control - AudioSilenceSmuggling: encodes instruction in pause-duration binary over short/long pause buckets; scores with existing ToolRiskActionDetector EPIC-3 — Multi-turn drift (TICKET 301) - PETTSDriftPoison: 5/20 turn sessions with persistent broadband noise on every benign turn; harmful request at the final turn; compares vs. clean control; sweeps session length as independent variable All probes skip gracefully (log warning, return []) on non-DuplexCapable targets. 268 tests, all passing.
1. RefusalReversalDetector: return None (not 0.0) for empty/whitespace full_transcript — blank output can't be distinguished from a safe refusal without this, an endpoint returning " " scores as a pass every time 2. DuplexPETTS.probe(): fix zip(scripts, audio_msgs) truncation bug — PETTSBargeInterrupt builds N_cases × N_affirmations scripts but audio_msgs has only N_cases entries; zip() silently dropped all but the first affirmation variant, turning 20 intended attempts into 5 3. PETTSBargeInterrupt: add StateDestabilizationDetector to extended_detectors so blank/degenerate responses surface as failures rather than vacuous passes in the harness summary (StateDestabilizationDetector scores empty→1.0, RefusalReversalDetector scores empty→None; both are now present) Also adds: - RefusalReversalDetector tests for empty/whitespace full_transcript → None - PETTSBargeInterrupt test asserting 5 cases × 4 affirmations = 20 scripts
Per reviewer feedback: generators should not silently perform modality conversion. If the duplex target returns audio but no text content (the cause of the blank-output results in the first live run), an optional explicit ASR fallback is now available via asr_uri / asr_model / asr_language config params. When asr_uri is set, NVDuplexChat calls NVAudioTranscription against the configured endpoint to produce a scorable transcript. When asr_uri is unset (default), blank output reaches detectors as-is so RefusalReversal returns None and StateDestabilization returns 1.0. The transcription step is user-visible in config, uses the same public default model as NVAudioTranscription (parakeet-1-1b-rnnt-multilingual), and is isolated in _transcribe_response_audio() so it can be tested independently. provenance["asr_used"] records whether the fallback fired.
Addresses reviewer guidance that (1) garak generators output text only and must not handle/transcribe non-text responses, and (2) new NIM generators should follow the upstream OpenAICompatible pattern like Vision / NVMultimodal. NVVoiceChat: - now extends NVOpenAIChat and talks to the target through the OpenAI SDK client (self.generator.create) instead of raw requests, matching the upstream nim generators - audio is sent as an input_audio content block via the inherited _conversation_to_list; the target's TEXT transcript is read from choices[0].message.content - REMOVED all audio-output handling: generate_audio response consumption, _save_response_audio, response_audio_dir, and the ASR fallback (asr_uri/asr_model/asr_language, _transcribe_response_audio). Transcribing the target's audio reply would make the test measure the target *as interpreted by a transcription provider*, not the target itself - keeps trailing-silence, system/text prompt, tools/tool_choice, and generate_audio-as-request-passthrough via a _prepare_prompt hook - request_timeout / request_retries now configure the SDK client (slow S2S endpoints need a long timeout; SDK handles backoff internally) NVDuplexChat: - run_session builds a growing Conversation and calls self._call_model per turn (SDK path) instead of custom HTTP; removed _send_turn, _post_completion, and the ASR fallback - text-out only, consistent with NVVoiceChat Tests rewritten to stub the SDK create call / _call_model rather than requests.post. Audio-output and ASR tests removed; added coverage for text-only output, empty-content passthrough, and no-audio-output-params.
Convention/correctness fixes found in a branch-wide audit against upstream: - detectors/audio_duplex.py: use lang_spec (upstream attr) instead of bcp47, which garak silently ignores. RefusalReversal and StateDestabilization detect() now return exactly one score per attempt.outputs entry (was max(len,1) / fixed [1.0], which could IndexError the evaluator when outputs is empty). Refusal signal is session-level, assigned per present output. - NVAudioTranscription: default uri moved off the NVIDIA-private inference-api.nvidia.com onto the public integrate.api.nvidia.com surface (the endpoint the reviewer flagged as internal-only); docstring notes the required transcription route so users can repoint if needed. - Removed AudioVoiceBleed probe + VoiceBleedDetector: the text-only generator refactor means agent audio is never captured, so the speaker-similarity detector could only ever return None. Speaker-identity bleed fundamentally needs audio output, which garak does not support; keeping inert code invites reviewer confusion. Can return as its own PR against an audio-capable path. Also dropped now-unused imports (attack metadata helpers, tempfile, _config).
NVAudioTranscription was scaffolding for the ASR fallback, which was removed
when the generators became text-out only. Nothing in the branch uses it, and
its /v1/audio/{model}/transcriptions route on the public endpoint is
unverified — shipping a target whose default may 404 is a quieter version of
the private-endpoint problem the reviewer flagged.
The reusable half of the original PR — the Generator.supported_formats()
format interface — stays. NVAudioTranscription can return as its own PR once
a public ASR endpoint + route is confirmed with a live test.
Removed the class + now-unused requests/mimetypes imports; renamed the test
file to test_nim_voicechat.py (it now covers only NVVoiceChat).
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Experimental integration branch that cherry-picks all six draft audio PRs onto current
mainso the full probe stack can be tested end-to-end against a live duplex S2S target before the individual PRs land upstream.What's included
pr/audio-generatorGenerator.supported_formats()interface +NVAudioTranscription(Parakeet ASR)pr/audio-resourcesgarak.resources.audio— TTS synthesis, WAV transforms with SHA-256 provenance, attack metadata helperspr/nvvoicechatNVVoiceChat— speech-to-speech generator via OpenAI-compat/v1/chat/completionsshimpr/audio-probesgarak/detectors/audio.py(output quality, tool-risk, reliability, judge)pr/audio-injectionAudioSuffixInjection+AudioOverlayInjectionprobespr/audio-bonPairedDirect+AcousticVoiceBestOfNBest-of-N probesTwo shim commits glue the stack onto
mainwithout pulling in the fullfeature/technique_intentbranch:garak/intents/+garak.services.intentservice— minimal stubs satisfyingIntentProbe's import (all concrete PETTS subclasses override the intent-population methods with hardcoded data anyway)_config.casnamespace —GarakSubConfig()entry expected by probe testsHow to test against the duplex endpoint
Status
pytest tests/generators/test_nim_audio_transcription.py tests/resources/ tests/detectors/test_detectors_audio*.py tests/probes/test_probes_audio*.py)mainThis branch is not intended for upstream merge — it exists purely for red-team testing. Once individual PRs land, switch back to running off their tips.