Audio: generator audio-input support (format interface + OpenAI-compatible audio) - #1
Closed
codingshreyash wants to merge 6 commits into
Closed
Audio: generator audio-input support (format interface + OpenAI-compatible audio)#1codingshreyash wants to merge 6 commits into
codingshreyash wants to merge 6 commits into
Conversation
Adds a format-capability interface to the generator base so probes can query which audio/image formats a target accepts and skip cleanly when a requested format is unsupported, instead of failing part-way through a run. Adds NVAudioTranscription, a generator for NVIDIA's transcription NIM, as the first consumer of the interface. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
- NVAudioTranscription now validates inline (raw-bytes) audio against audio_formats using the mime the message carries, and forwards the real content type, instead of labelling every payload as WAV. - OpenAICompatible.audio_formats is derived from audio_mime_subtype_formats' values so supported_formats() cannot advertise a format the request builder is unable to send. Adds tests for both paths. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
This was referenced Jul 21, 2026
Per review: generators abstract communication to the target and return text; an ASR-target generator with a private-endpoint default doesn't fit. This PR now carries only the generator audio-format interface and OpenAI-compatible audio input support that the audio probes and the VoiceChat target rely on. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
codingshreyash
changed the title
The parametrized multiprocessing test drives a text prompt; audio-only targets (e.g. NVVoiceChat) cannot process it, so filter them out by input modality. Signed-off-by: Shreyash Ranjan <shrranjan@nvidia.com>
Owner
Author
|
Closing as superseded. NVIDIA#1770 now supplies the generator-format/PETTS foundation, while the remaining provider/provenance work is isolated in #2 and the target adapter in #3. The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the generator-side support the audio probes need to send audio to a target:
Generator(supported_formats()+ per-modality format sets) so probes can check what a target accepts and skip cleanly when a format is unsupported.OpenAIAudioCompatiblebase for endpoints that acceptinput_audiocontent.No generators are added here; the target-under-test generator (VoiceChat) lands in a follow-up that builds on this. Output stays text-only per garak convention.
Verification
python -m pytest tests/generators/test_openai_compatible.py(Fork review draft; upstream target
feature/technique_intent.)