Skip to content

Add portable Qwen CPU/CUDA/ROCm backends alongside MLX - #95

Merged
phoebus-84 merged 7 commits into
mainfrom
feat/portable-tts-backends
Nov 16, 2026
Merged

phoebus-84 merged 7 commits into
mainfrom
feat/portable-tts-backends

Conversation

@phoebus-84

@phoebus-84 phoebus-84 commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

Extend video-studio/packages/voice, not the website or an independent TTS repository. Based on the latest main at the start of work (3d51988), developed in a separate worktree; the operator's existing checkout/branch and changes were not switched, reset or stashed.

Backend choice

  • Keep MLX/Metal and its existing quantized checkpoints on Apple Silicon.
  • Add the official Qwen/PyTorch engine for CPU, NVIDIA CUDA and supported AMD ROCm.
  • Explicit backend/device selection, capability-based GPU precision, SDPA by default, optional FA2 and CPU thread/BF16 tuning. Unsupported requests fail instead of silently changing device/provider or producing fake success.
  • Reuse the API, warm-model lifecycle, consent-gated cloning, canonical WAV validation and durable attempt infrastructure.

GGUF implementations were inspected, but are not advertised as implemented. Their complete speech pipeline, profile/control parity, native artifacts and runtime baseline need an independently pinned/benchmarked adapter. This chooses broad official-model compatibility, not an unsubstantiated “fastest everywhere” claim.

Integrity / provenance

  • Five pinned official Torch size/profile snapshots with exact commit revisions and critical SHA-256 values, in distinct *-torch directories. Existing *-8bit installs are not overwritten or reused as Torch models.
  • Resolved runtime options/versions are recorded on successful responses and portable durable profiles, included in cache identities, and optionally preserved by the narration schema/TS bindings.
  • Historical MLX fingerprints without runtime metadata retain their prior calculation.
  • Generation/seed state is serialized; Torch uses inference mode and restores RNG state. Warm leases still prevent unloading active models.
  • Selected-runtime diagnostics, CLI options, backend guide and ADR-0051 (renumbered during integration so films keep ADR-0050).

Validation completed locally

  • 422 Python tests passed, 6 opt-in tests skipped.
  • npm run check passed across the monorepo (21 module test runs, zero failed).
  • npm run check:contracts passed after adding runtime provenance (416 tests).
  • npm run docs:build passed.
  • Built the Python wheel and checked that it contains the pinned Torch catalogue.
  • Actual offline CPU synthesis in English and Italian, non-silent 24 kHz mono PCM16, two requests sharing one model load.
  • Actual offline Base cloning using only a newly generated synthetic preset reference, consent record and integrity checks; the private reference was revoked afterwards.
  • Local live inference was CPU-only on macOS ARM64 with PyTorch 2.15.0 / Qwen TTS 0.1.1. It is a functional smoke, not a controlled throughput or listening-quality benchmark. No audio/model weights are committed or published.

Explicit limitations

  • ROCm hardware acceptance now passed on RX 7900 XTX / Torch 2.11.0+rocm7.2 / Qwen TTS 0.1.1, BF16 and SDPA, in an isolated environment on 2026-11-16. 422 Python tests and real offline preset + consented synthetic-only cloning passed; revocation was checked. Longer Italian preset and clone samples transcribed with zero normalized word errors. Cold/first-use timings are in packages/voice/docs/backends.md; these are not a controlled throughput benchmark or listening-quality approval. NVIDIA CUDA hardware remains untested.
  • Integration after Films: self-contained works produced at three levels, with a cut editor #99: npm run check passed (22 workspace test runs), docs build passed, studio synthesis receipts preserve worker runtime and explicit preset seeds are forwarded. CI, including the opt-in Linux CPU live smoke, passed.
  • Forced alignment, transcription and optional ASR diagnostics remain MLX-only, explicitly reported as such. Portable TTS does not imply those features are portable.
  • VoiceDesign remains install/catalogue-only, as previously; this PR implements preset synthesis and consented Base cloning.
  • Windows foreground operation is supported by the portable path, but LaunchAgent is macOS-only and POSIX permission tests do not prove Windows ACLs.

CI now runs real hardware-independent tests on Linux/macOS, portable adapter/config/lifecycle/contracts/synthesis tests on Windows. Apply voice-live-smoke to explicitly opt into the pinned public-model Linux x64 CPU live smoke (or dispatch the workflow manually after it is on main); it uploads JSON measurements only.

@phoebus-84 phoebus-84 added the voice-live-smoke Opt in to pinned public-model local CPU speech synthesis checks label Nov 11, 2026
@phoebus-84

Copy link
Copy Markdown
Owner Author

Linux x64 actual CPU synthesis passed: live run 43022955802. English and Italian, offline, official pinned 0.6B CustomVoice, float32/SDPA, 2 CPU threads; one warm-model load. The uploaded JSON records timing and hashes only. Warm Italian request: 10.65 s for 2.64 s audio (RTF 4.04), so this is deliberately not a real-time CPU claim. Linux/macOS Python suites also passed. Windows initially caught two pre-existing POSIX chmod assertions; these now assert permission bits only on POSIX while still checking directory creation everywhere. The ADR now declares its proposed status for the docs checker.

@phoebus-84

Copy link
Copy Markdown
Owner Author

Final validation on head db05de6: CI passed, including the Linux/macOS Python suites, Windows portable tests, documentation, invariants and all existing module jobs: https://github.com/phoebus-84/video-studio/actions/runs/43028839487 . The opt-in actual Linux x64 CPU synthesis also passed again: https://github.com/phoebus-84/video-studio/actions/runs/43028839825 . CUDA/ROCm performance remains explicitly unmeasured; no GGUF backend or portable alignment/ASR is claimed. The isolated worktree is clean, and this PR has not been merged.

@phoebus-84

Copy link
Copy Markdown
Owner Author

Real hardware validation completed on cecino without modifying/restarting ComfyUI: RX 7900 XTX, Python 3.12.16, Torch 2.11.0+rocm7.2, qwen-tts 0.1.1, cuda:0/BF16/SDPA, offline official 1.7B weights. 422 Python tests passed (6 opt-in skipped); live English/Italian smoke passed with one warm model load. Extended Italian preset: 10.80 s audio / 24.48 s wall; synthetic-only consented Base clone: 6.40 s / 71.29 s including first-use costs; deleting the enrolled profile blocked further synthesis. Both longer samples transcribed offline by faster-whisper large-v3 with zero normalized word errors. No listening/voice-similarity approval or controlled performance claim. Reports/logs and synthetic WAVs remain in the isolated ~/ai/video-studio-pr95-smoke/ directory, never git. Integration fixes preserve runtime in studio synthesis receipts and pass explicit preset seeds through; ADR renumbered to 0051 so film keeps 0050.

@phoebus-84
phoebus-84 merged commit e908156 into main Nov 16, 2026
53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

voice-live-smoke Opt in to pinned public-model local CPU speech synthesis checks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant