Skip to content

Audio: spoken-channel reliability precursors - #4

Closed
codingshreyash wants to merge 1 commit into
pr/audio-resourcesfrom
pr/audio-probes
Closed

Audio: spoken-channel reliability precursors#4
codingshreyash wants to merge 1 commit into
pr/audio-resourcesfrom
pr/audio-probes

Conversation

@codingshreyash

@codingshreyash codingshreyash commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Summary

Adds two inactive, benign calibration probes for the PETTS audio path:

  • TTSReliability checks whether short words and symbol boundaries survive synthesis and the target's audio-input path.
  • SemanticReliability checks whether simple spoken requests retain their meaning.

Each probe has a dedicated detector and explicit attempt metadata. Missing target text remains unscored; it is not counted as a safe response. This PR contains no tool-risk payloads, overlays, ASR service, or model-as-judge.

Relationship to other work

The upstream duplicate search found NVIDIA#1770 for the underlying PETTS technique and no open PR for these reliability precursors. This PR is a focused child of the provider/provenance refactor in #2.

Verification

Run locally with Python 3.12:

XDG_CACHE_HOME=/private/tmp/garak-pr-audit-cache \
XDG_DATA_HOME=/private/tmp/garak-pr-audit-data \
python -m pytest -q \
  tests/probes/test_probes_audio_tts_reliability.py \
  tests/probes/test_probes_audio_semantic_reliability.py \
  tests/detectors/test_detectors_audio_tts_reliability.py \
  tests/detectors/test_detectors_audio_semantic_reliability.py \
  tests/test_docs.py

Result: 695 passed.

python -m black --check <changed Python files>
git diff --check

Result: clean.

No new dependency is added.

AI assistance

ChatGPT was used to review this change.

@codingshreyash codingshreyash changed the title Audio: PETTS probe family + tool-risk/reliability detectors Audio: spoken-channel reliability precursors Jul 23, 2026
@codingshreyash
codingshreyash changed the base branch from base/probes to pr/audio-resources July 23, 2026 21:08
Calibrate exact transcription and simple spoken-request comprehension before interpreting adversarial audio results.

AI-Review: ChatGPT
@codingshreyash

Copy link
Copy Markdown
Owner Author

Superseded by the rebased, cleaned 3-PR stack based on current feature/technique_intent: #9 (foundation), #10 (probes+detectors), #11 (injection). Content preserved there; closing to consolidate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant