Skip to content

feat(agent): speech through a ComfyUI server's Qwen3-TTS nodes, with cloning and clean-then-clone - #98

Merged
phoebus-84 merged 6 commits into
mainfrom
feat/comfyui-qwen-speech
Nov 16, 2026
Merged

phoebus-84 merged 6 commits into
mainfrom
feat/comfyui-qwen-speech

Conversation

@aigeneratoralpha

@aigeneratoralpha aigeneratoralpha commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Optional ComfyUI Qwen3-TTS speech behind the existing pva_synthesize_speech tool. Integrated after #99 and #95; the local Qwen worker (MLX or Torch) remains the default and failures never switch provider.

Originally stacked on #96. This branch now targets main, carrying only the two ComfyUI commits and integration fixes: #96's ElevenLabs/avatar features are not included.

Features

  • New reusable @av/comfyui-speech: explicit server configuration, queue/poll/upload/download client, node/readiness probing and five API-format JSON workflows: preset speaker, reference clone, saved voice, clean-then-clone and clean-only.
  • provider: "comfyui-qwen" uses the same episode/source/film/scene/shot workspace addressing and synthesis → register → align → verify flow. No fallback; no change to the worker's durable attempt/mastering contracts or sealed series profiles.
  • pva_clean_voice separates vocals from an optional range into a new workspace asset, without cloning.
  • Reference and saved-voice clones require voiceConsentPath, an operator-created JSON authorization record bound to the reference SHA-256 or exact saved:<name>. Checked before upload/execution; removing the record or setting confirmed: false blocks subsequent calls. The agent must not grant consent for the operator.
  • Speech output is validated non-silent mono PCM16/24 kHz WAV. Clean-only keeps the separation's native sample rate/channels. FFmpeg conversion is bounded and cancellable.
  • Receipts preserve workflow hash, consent-record hash, reference hashes, seed and requested settings. runtimeVerified: false explicitly acknowledges that ComfyUI does not attest actual checkpoint hashes/resolved runtime versions. Saved voice names are likewise not immutable waveform identities.
  • Running graphs are never interrupted globally; cancellation removes queued work when possible. Uploads and optionally saved voices remain on ComfyUI; local consent revocation does not remotely delete them.
  • Narration, film-sound and executive-producer skills document the optional path. The new package has its own CI matrix entry. Fixed the README directory link that broke the docs build.

Validation

  • npm run check passed: 23 workspace test runs, zero failures.
  • npm run docs:build passed, full profile composition passed; agent typecheck passed.
  • Package: 25 tests. Focused agent suites (ComfyUI, local preset, local clone, tool registration): 43 tests, including all three film levels and channel sources, consent mismatch/revocation, no fallback, stereo/44.1 kHz normalization and rejection of silent/corrupt speech.
  • Live adapter smoke on the existing ComfyUI server on cecino (RX 7900 XTX/ROCm), without modifying/restarting its installation. Only freshly generated synthetic references were uploaded; existing human recordings/saved voices were not used or changed.
Live operation Audio Adapter elapsed
Italian preset 4.697 s 8.034 s
Synthetic reference clone 4.793 s 7.979 s
Clean then synthetic clone 3.994 s 7.980 s
Clean-only, 0.5 s + 4 s range 4.000 s 3.968 s

Offline faster-whisper large-v3 transcribed both clone samples with zero normalized word errors. The preset differed only in spelling of the product name (ComfyUI → ConfiUi, 1/9 words). This is a technical/content smoke, not a listening or voice-similarity approval or a controlled comparison against #95.

Reports, logs and synthetic audio remain under ~/ai/video-studio-pr98-smoke/, never git. The smoke wrote only its own studio-pr98-smoke ComfyUI input/output namespace; it did not save a new speaker.

Configuration and limits

Set AVG_COMFYUI_URL, or url in library/.render/comfyui.json (AVG_COMFYUI_CONFIG can name another file). For loopback-only servers, open your own SSH tunnel. Node settings include device, precision, attention and unloadModelAfterGenerate (default true).

No word timings, VoiceDesign, dialogue/fine-tuning nodes or automatic tunnel management. Saved-voice execution/save-as remain fake-server-tested only; no existing saved voice was exercised live. Full consent-record format and provenance limits are in docs/running.md.

aigeneratoralpha and others added 6 commits November 16, 2026 17:51
…Qwen3-TTS nodes

@av/comfyui-speech is a small ComfyUI client (queue, wait on the history,
upload an input, download an output) and four workflows kept as API-format
templates: a preset speaker, a cloned reference, a voice saved on the server,
and cleaning a reference (cut, voice separation) before cloning it.

pva_synthesize_speech takes provider "comfyui-qwen": voice is a speaker or
saved:<name>, referenceAudioPath clones a recording of the workspace, and
cleanReference cleans it first. It is free, never chosen on its own, and
writes the WAV and its synthesis record like the other providers. The
server's address is AVG_COMFYUI_URL or the library's .render/comfyui.json.
pva_capability_status says whether the server answers and what it can run;
pva_voices lists its speakers and saved voices.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…fyUI server, without cloning

@av/comfyui-speech gains cleanVoice and workflows/clean.json: LoadAudio, an
optional TrimAudioDuration (bypassed when no cut is asked for), AudioSeparation's
Vocals, SaveAudio, with the clean-clone chain's nodes and inputs. The new
pva_clean_voice keeps the voice as a new WAV under the workspace's assets/,
with <file>.clean.json beside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@phoebus-84
phoebus-84 force-pushed the feat/comfyui-qwen-speech branch from 1673fff to 57ddce9 Compare November 16, 2026 17:11
@phoebus-84
phoebus-84 changed the base branch from feat/elevenlabs-speech-avatar to main November 16, 2026 17:11
@phoebus-84

Copy link
Copy Markdown
Owner

Integration is now rebased/transplanted onto main after #99 → #95, without #96. Original head is preserved locally as refs/backup/pr98-before-integration (1673fff). Full local npm run check passed (23 workspace test runs), docs build passed, and live adapter tests on cecino passed for preset, synthetic-only clone, clean-then-clone and clean-only. Consent mismatch/revocation, canonical WAV/silence checks, and episode/source/all three film levels are covered. See updated PR body for measurements and limits. Waiting for the updated CI before merging.

@phoebus-84
phoebus-84 merged commit f927774 into main Nov 16, 2026
54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants