Skip to content

feat(agent): a photo speaks (H3 Max lip-sync); ElevenLabs speech keeps its word timings - #96

Open
aigeneratoralpha wants to merge 2 commits into
mainfrom
feat/elevenlabs-speech-avatar
Open

aigeneratoralpha wants to merge 2 commits into
mainfrom
feat/elevenlabs-speech-avatar

Conversation

@aigeneratoralpha

@aigeneratoralpha aigeneratoralpha commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

What this adds

A talking avatar from one photo. pva_generate_avatar_speech takes a photo and 5–60 s of audio and returns a talking clip through MiniMax H3 Max lip-sync on fal.

  • The video is as long as the audio; the price is per second by resolution (480P $0.05, 768P $0.08, 1080P $0.16, 2K $0.32), ×1.2 for the whole clip when it is longer than 15 s.
  • It goes through the episode's spend ledger and writes assets/video/<shotId>.mp4 with a provenance sidecar.
  • imageSource selects a shared channel source; imagePath is relative to it, audioPath stays episode-relative.

ElevenLabs speech, chosen explicitly. pva_synthesize_speech takes provider: "elevenlabs" with a voiceId. Qwen stays the default; there is no automatic fallback between providers.

  • @av/elevenlabs gains synthesizeSpeech (/v1/text-to-speech/{voice_id}/with-timestamps): PCM wrapped in WAV, character timestamps turned into words.
  • The tool writes <sceneId>.synthesis.json beside the WAV: what made it, and the words it returned.

Word timings without a second paid call. pva_align_take takes provider: "synthesis": it finds the synthesis record by the audio's hash and reuses its words. No aligner runs.

Smaller changes

  • @av/fal-media: H3 Max Turbo image-to-video profile; lip-sync client, types and prices.
  • pva_capability_status reports the two new capabilities.
  • tools/generate-h3-from-motion-plan.mjs: fal's current parameter names (end_image_url, prompt_expansion_mode).
  • The executive producer skill describes the two tools.

How it was brought here

The work was uncommitted on an old checkout of feat/render-worker (44 commits behind main). It was moved onto current main; three files conflicted and keep both sides:

  • studio/agent/extensions/index.ts: FLUX 3 Max and the avatar tools are both registered.
  • studio/agent/tests/extension.test.ts: 71 tools (70 + pva_generate_avatar_speech).
  • studio/agent/skills/pva-executive-producer/SKILL.md: main's text, plus the two tool entries. The lines that named a channel and the operator's machines were not brought back.

Tests

Run on Windows:

  • tsc --noEmit: passes in packages/elevenlabs, packages/fal-media, studio/episode, studio/agent.
  • packages/elevenlabs: 24 of 24. packages/fal-media: 170 of 170.
  • studio/agent, the three files this touches (avatar-tools, elevenlabs-speech, extension): 43 of 43.
  • studio/episode take-align.test.ts: 9 of 10. The one failure is "refuses a take's audio that is a link out of the episode", which needs a symbolic link this Windows account cannot create.
  • npm run check:library-names: passes.

Not run to the end here: npm run check stops at @studio/contracts, where 6 tests fail on Windows only (file modes, symbolic links, path separators), in files this PR does not touch. The full suites of studio/agent and studio/episode have further Windows-only failures of the same kind and 5 s timeouts. CI on Linux is the full check.

No paid request was made for this PR: the provider calls are tested against a faked fetch.

🤖 Generated with Claude Code

aigeneratoralpha and others added 2 commits November 14, 2026 19:23
…s its word timings

- packages/elevenlabs: synthesizeSpeech (text-to-speech with timestamps),
  PCM wrapped in WAV, character timings turned into words; profile with
  defaults and price estimates.
- packages/fal-media: MiniMax H3 Max lip-sync (image + audio to video),
  priced per second by resolution with the long-video surcharge; H3 Max
  Turbo image-to-video profile.
- studio/agent: pva_generate_avatar_speech (photo + 5-60 s audio to a
  talking clip, on the spend ledger, with a provenance sidecar);
  pva_synthesize_speech takes provider "elevenlabs" with a voiceId and
  writes <sceneId>.synthesis.json beside the WAV; capability status
  reports both.
- studio/episode: pva_align_take provider "synthesis" reads the words the
  speech model returned, by the audio's hash: no aligner, no second paid
  call.
- tools/generate-h3-from-motion-plan: fal's current parameter names
  (end_image_url, prompt_expansion_mode).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant