Repository navigation
feat(agent): a photo speaks (H3 Max lip-sync); ElevenLabs speech keeps its word timings - #96
Open
aigeneratoralpha wants to merge 2 commits into
Open
aigeneratoralpha wants to merge 2 commits into
aigeneratoralpha wants to merge 2 commits into
Conversation
…s its word timings - packages/elevenlabs: synthesizeSpeech (text-to-speech with timestamps), PCM wrapped in WAV, character timings turned into words; profile with defaults and price estimates. - packages/fal-media: MiniMax H3 Max lip-sync (image + audio to video), priced per second by resolution with the long-video surcharge; H3 Max Turbo image-to-video profile. - studio/agent: pva_generate_avatar_speech (photo + 5-60 s audio to a talking clip, on the spend ledger, with a provenance sidecar); pva_synthesize_speech takes provider "elevenlabs" with a voiceId and writes <sceneId>.synthesis.json beside the WAV; capability status reports both. - studio/episode: pva_align_take provider "synthesis" reads the words the speech model returned, by the audio's hash: no aligner, no second paid call. - tools/generate-h3-from-motion-plan: fal's current parameter names (end_image_url, prompt_expansion_mode). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A talking avatar from one photo.
pva_generate_avatar_speechtakes a photo and 5–60 s of audio and returns a talking clip through MiniMax H3 Max lip-sync on fal.assets/video/<shotId>.mp4with a provenance sidecar.imageSourceselects a shared channel source;imagePathis relative to it,audioPathstays episode-relative.ElevenLabs speech, chosen explicitly.
pva_synthesize_speechtakesprovider: "elevenlabs"with avoiceId. Qwen stays the default; there is no automatic fallback between providers.@av/elevenlabsgainssynthesizeSpeech(/v1/text-to-speech/{voice_id}/with-timestamps): PCM wrapped in WAV, character timestamps turned into words.<sceneId>.synthesis.jsonbeside the WAV: what made it, and the words it returned.Word timings without a second paid call.
pva_align_taketakesprovider: "synthesis": it finds the synthesis record by the audio's hash and reuses its words. No aligner runs.Smaller changes
@av/fal-media: H3 Max Turbo image-to-video profile; lip-sync client, types and prices.pva_capability_statusreports the two new capabilities.tools/generate-h3-from-motion-plan.mjs: fal's current parameter names (end_image_url,prompt_expansion_mode).How it was brought here
The work was uncommitted on an old checkout of
feat/render-worker(44 commits behindmain). It was moved onto currentmain; three files conflicted and keep both sides:studio/agent/extensions/index.ts: FLUX 3 Max and the avatar tools are both registered.studio/agent/tests/extension.test.ts: 71 tools (70 +pva_generate_avatar_speech).studio/agent/skills/pva-executive-producer/SKILL.md:main's text, plus the two tool entries. The lines that named a channel and the operator's machines were not brought back.Tests
Run on Windows:
tsc --noEmit: passes inpackages/elevenlabs,packages/fal-media,studio/episode,studio/agent.packages/elevenlabs: 24 of 24.packages/fal-media: 170 of 170.studio/agent, the three files this touches (avatar-tools,elevenlabs-speech,extension): 43 of 43.studio/episodetake-align.test.ts: 9 of 10. The one failure is "refuses a take's audio that is a link out of the episode", which needs a symbolic link this Windows account cannot create.npm run check:library-names: passes.Not run to the end here:
npm run checkstops at@studio/contracts, where 6 tests fail on Windows only (file modes, symbolic links, path separators), in files this PR does not touch. The full suites ofstudio/agentandstudio/episodehave further Windows-only failures of the same kind and 5 s timeouts. CI on Linux is the full check.No paid request was made for this PR: the provider calls are tested against a faked fetch.
🤖 Generated with Claude Code