From a0d383c01c080d10876ee00acf6fbcab86fe663a Mon Sep 17 00:00:00 2001 From: tigranbs Date: Thu, 21 May 2026 12:44:45 -0700 Subject: [PATCH 1/5] feat(ws): add loading-indicator audio loop on a dedicated LiveKit track Add a loading-indicator audio capability to the WebSocket API: a short client-supplied clip looped into the LiveKit room while the calling application is busy, so the human participant hears "still working" instead of silence. - config: new optional `loading_audio` object (base64 WAV or raw PCM with format/sample_rate/channels/volume); decoded and validated once per session, non-fatal on failure. - new `loading_start` / `loading_stop` WebSocket commands, fire-and-forget. - publish a second, dedicated `loading-audio` LiveKit track, independent of the `tts-audio` speech track; the loading loop never touches the TTS operation queue, audio queue, or generation state. - seamless cursor-based looping with a short linear fade-out on stop. - lifecycle handling for connect, reconnect, and disconnect teardown. The loop is controlled exclusively by `loading_start` / `loading_stop`; `speak` and `clear` are unaffected. No new dependencies. --- CLAUDE.md | 3 +- docs/api-reference.md | 31 +- docs/livekit_integration.md | 36 +- docs/openapi.yaml | 61 +++ docs/websocket.md | 187 ++++++- src/docs/openapi.rs | 5 +- src/handlers/ws/config.rs | 36 ++ src/handlers/ws/config_handler.rs | 51 +- src/handlers/ws/loading_handler.rs | 191 +++++++ src/handlers/ws/messages.rs | 115 +++- src/handlers/ws/mod.rs | 3 + src/handlers/ws/processor.rs | 14 +- src/handlers/ws/tests.rs | 141 ++++- src/livekit/client/audio.rs | 37 ++ src/livekit/client/connection.rs | 23 + src/livekit/client/loading_audio.rs | 562 ++++++++++++++++++++ src/livekit/client/mod.rs | 62 +++ src/livekit/client/operation_worker.rs | 20 + src/livekit/client/tests.rs | 266 ++++++++++ src/livekit/loading_clip.rs | 704 +++++++++++++++++++++++++ src/livekit/mod.rs | 2 + 21 files changed, 2523 insertions(+), 27 deletions(-) create mode 100644 src/handlers/ws/loading_handler.rs create mode 100644 src/livekit/client/loading_audio.rs create mode 100644 src/livekit/loading_clip.rs diff --git a/CLAUDE.md b/CLAUDE.md index b30b740..80907be 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -225,9 +225,10 @@ See existing implementations for patterns: ### WebSocket - `/ws` - Real-time voice processing - - Receives: Config, audio data, speak commands, **clear command**, send_message, sip_transfer + - Receives: Config (optionally with `loading_audio`), audio data, speak commands, **clear command**, loading_start, loading_stop, send_message, sip_transfer - Sends: Ready, STT results, TTS audio, messages, participant_connected, participant_disconnected, track_subscribed, tts_playback_complete, vad_event, error, sip_transfer_error - **Clear command**: Immediately stops TTS and clears audio buffers (fire-and-forget, respects `allow_interruption` setting) + - **Loading commands**: `loading_start`/`loading_stop` control a looping loading-indicator audio clip on a dedicated `"loading-audio"` LiveKit track (fire-and-forget, independent of `speak`/`clear`) ### Webhooks - `POST /livekit/webhook` - LiveKit event receiver (signature verified) diff --git a/docs/api-reference.md b/docs/api-reference.md index 1d01164..db866ef 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -544,7 +544,7 @@ This enables efficient trunk management without manual provisioning. 1. Connect via WebSocket and immediately send a `config` message. 2. The server initializes providers (and LiveKit, if requested) and replies with `ready`. 3. Stream audio frames as binary messages that match the declared STT sample rate/encoding. -4. Use `speak`, `clear`, or `send_message` commands to drive TTS and LiveKit data. +4. Use `speak`, `clear`, `loading_start`, `loading_stop`, or `send_message` commands to drive TTS, the loading indicator, and LiveKit data. 5. Close the socket when finished; the server also closes when a fatal `error` is emitted. #### Incoming Messages @@ -560,6 +560,7 @@ Configures audio processing and optional LiveKit mirroring. Must be the first me | `stt_config` | object | Conditional | Required when `audio=true`. See table below. | | `tts_config` | object | Conditional | Required when `audio=true`. Same schema as the REST `tts_config`. | | `livekit` | object | No | LiveKit options; omitted for WebSocket-only sessions. | +| `loading_audio` | object | No | Optional loading indicator clip. Processed only when `audio=true` and `livekit` is present; a decode failure is non-fatal. See table below. | **STT configuration** @@ -583,6 +584,20 @@ Configures audio processing and optional LiveKit mirroring. Must be the first me | `sayna_participant_name` | string | Override for the agent display name (default `Sayna AI`). | | `listen_participants` | array | Restrict audio/data processing to specific participant identities (empty list listens to all). | +**`loading_audio` configuration** + +Decoded once at config time; the validated clip is held for the session and played on a dedicated `"loading-audio"` LiveKit track via the `loading_start` / `loading_stop` commands. Only 16-bit PCM is supported. + +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| `data` | string | Yes | Base64-encoded audio bytes: a complete WAV file or raw PCM. | +| `format` | string | No | `wav` or `pcm`. Auto-detected from the `RIFF`/`WAVE` signature when omitted. | +| `sample_rate` | integer | Conditional | Sample rate in Hz (8000–48000). Required for raw PCM; ignored for WAV. | +| `channels` | integer | No | Channel count for raw PCM (`1` or `2`, default `1`); ignored for WAV. | +| `volume` | number | No | Playback volume `0.0`–`1.0` (default `1.0`); out-of-range values are clamped. | + +See [Loading Indicator](websocket.md#loading-indicator) in the WebSocket reference for authoring guidance and limits. + ##### `speak` Queues text for synthesis. @@ -600,6 +615,20 @@ Immediately clears queued audio and LiveKit buffers. Useful for interruptions. | --- | --- | --- | | `type` | string | `clear`. | +##### `loading_start` +Begins looping the configured loading indicator clip on a dedicated `"loading-audio"` LiveKit track. Requires `audio=true`, an active LiveKit room, and a `loading_audio` clip supplied in `config`. Fire-and-forget on success; idempotent if the loop is already running; failures emit an `error` message. + +| Field | Type | Description | +| --- | --- | --- | +| `type` | string | `loading_start`. | + +##### `loading_stop` +Stops the loading indicator loop with a short fade-out. Fire-and-forget; idempotent and always silent — stopping when nothing is playing is never an error. The loop is controlled only by `loading_start` / `loading_stop`; `speak` and `clear` do not affect it. + +| Field | Type | Description | +| --- | --- | --- | +| `type` | string | `loading_stop`. | + ##### `send_message` Publishes a LiveKit data message (if LiveKit is configured) and also feeds the VoiceManager message bus. diff --git a/docs/livekit_integration.md b/docs/livekit_integration.md index 85e1123..f753f10 100644 --- a/docs/livekit_integration.md +++ b/docs/livekit_integration.md @@ -17,15 +17,16 @@ Sayna provides comprehensive LiveKit integration with automatic SIP infrastructu ## Table of Contents 1. [Architecture Overview](#architecture-overview) -2. [Configuration](#configuration) -3. [Inbound Webhooks (LiveKit → Sayna)](#inbound-webhooks-livekit--sayna) -4. [SIP Configuration & Auto-Provisioning](#sip-configuration--auto-provisioning) -5. [Outbound SIP Calls](#outbound-sip-calls) -6. [Outbound Webhooks (Sayna → Downstream)](#outbound-webhooks-sayna--downstream) -7. [Webhook Signing](#webhook-signing) -8. [Testing & Development](#testing--development) -9. [Operations & Troubleshooting](#operations--troubleshooting) -10. [Security Considerations](#security-considerations) +2. [Published Audio Tracks](#published-audio-tracks) +3. [Configuration](#configuration) +4. [Inbound Webhooks (LiveKit → Sayna)](#inbound-webhooks-livekit--sayna) +5. [SIP Configuration & Auto-Provisioning](#sip-configuration--auto-provisioning) +6. [Outbound SIP Calls](#outbound-sip-calls) +7. [Outbound Webhooks (Sayna → Downstream)](#outbound-webhooks-sayna--downstream) +8. [Webhook Signing](#webhook-signing) +9. [Testing & Development](#testing--development) +10. [Operations & Troubleshooting](#operations--troubleshooting) +11. [Security Considerations](#security-considerations) --- @@ -91,6 +92,23 @@ Sayna provides comprehensive LiveKit integration with automatic SIP infrastructu --- +## Published Audio Tracks + +When a WebSocket session is audio-enabled and joins a LiveKit room, the Sayna agent participant publishes synthesized speech on a single audio track named `"tts-audio"`. + +If that session's `config` message also includes a `loading_audio` object (see [Loading Indicator](websocket.md#loading-indicator) in the WebSocket reference), the agent participant publishes a **second** audio track named `"loading-audio"`, carrying the loading indicator sound. In that case the agent publishes **two** audio tracks: + +| Track name | Carries | +|------------|---------| +| `"tts-audio"` | Synthesized speech (text-to-speech output). | +| `"loading-audio"` | The loading indicator clip, looped while the calling application is busy. | + +The two tracks are independent and can be audible at the same time. LiveKit client SDKs automatically play every subscribed audio track, so participants hear both without any extra client-side handling. + +**Recordings:** Because `"loading-audio"` is a real published track, it is included in room-composite egress recordings alongside `"tts-audio"`. This is correct behavior — the recording faithfully reflects what the human participant heard. + +--- + ## Configuration ### Required Environment Variables diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 4b85b42..3e41dd4 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -661,6 +661,11 @@ components: - type: 'null' - $ref: '#/components/schemas/LiveKitWebSocketConfig' description: Optional LiveKit configuration for real-time audio streaming + loading_audio: + oneOf: + - type: 'null' + - $ref: '#/components/schemas/LoadingAudioConfig' + description: Optional loading-indicator audio configuration. stream_id: type: - string @@ -791,6 +796,24 @@ components: - type: 'null' - $ref: '#/components/schemas/VADConfigUpdate' description: VAD configuration updates (silence threshold, etc.) + - type: object + description: Begin looping the configured loading-indicator audio into the LiveKit room. + required: + - type + properties: + type: + type: string + enum: + - loading_start + - type: object + description: Stop the loading-indicator audio loop (with a short fade-out). + required: + - type + properties: + type: + type: string + enum: + - loading_stop description: WebSocket message types for incoming messages ListRoomsResponse: type: object @@ -858,6 +881,44 @@ components: - 'null' description: Sayna AI participant display name (defaults to "Sayna AI") example: Sayna AI + LoadingAudioConfig: + type: object + description: Loading-indicator audio configuration supplied in the `config` message. + required: + - data + properties: + channels: + type: + - integer + - 'null' + format: int32 + description: Channel count for raw PCM (1 = mono, 2 = stereo); defaults to 1. Ignored for WAV. + example: 1 + minimum: 0 + data: + type: string + description: Base64-encoded audio bytes — a complete WAV file or raw 16-bit PCM. + format: + type: + - string + - 'null' + description: 'Audio format: "wav" or "pcm". If omitted, the server auto-detects.' + example: wav + sample_rate: + type: + - integer + - 'null' + format: int32 + description: Sample rate in Hz. Required for raw PCM; ignored for WAV (header is authoritative). + example: 16000 + minimum: 0 + volume: + type: + - number + - 'null' + format: float + description: Playback volume from 0.0 (silent) to 1.0 (authored level). Default 1.0; out-of-range clamped. + example: 0.3 MuteParticipantRequest: type: object description: |- diff --git a/docs/websocket.md b/docs/websocket.md index 042b34c..a5c9aca 100644 --- a/docs/websocket.md +++ b/docs/websocket.md @@ -13,12 +13,13 @@ This guide walks you through everything you need to integrate Sayna's WebSocket 3. [Session Lifecycle](#session-lifecycle) 4. [Message Protocol](#message-protocol) 5. [Configuration](#configuration) -6. [Use Case Patterns](#use-case-patterns) -7. [Audio Processing](#audio-processing) -8. [LiveKit Integration](#livekit-integration) -9. [Error Handling](#error-handling) -10. [Best Practices](#best-practices) -11. [Troubleshooting](#troubleshooting) +6. [Loading Indicator](#loading-indicator) +7. [Use Case Patterns](#use-case-patterns) +8. [Audio Processing](#audio-processing) +9. [LiveKit Integration](#livekit-integration) +10. [Error Handling](#error-handling) +11. [Best Practices](#best-practices) +12. [Troubleshooting](#troubleshooting) --- @@ -199,6 +200,7 @@ This is where your application logic runs. You can: - Stream binary audio frames for transcription - Send `speak` commands for text-to-speech - Send `clear` commands to interrupt TTS playback +- Send `loading_start` / `loading_stop` commands to control the loading indicator audio - Send `send_message` commands for LiveKit data channels - Receive STT transcription results - Receive TTS audio chunks @@ -421,7 +423,77 @@ If `allow_interruption: false` was set on the most recent speak command, the cle --- -#### 5. Send Message +#### 5. Loading Start Message + +**Purpose:** Begin looping the configured loading indicator audio into the LiveKit room. + +**Structure:** +```json +{ + "type": "loading_start" +} +``` + +**Behavior:** +- Starts the configured loading clip on a continuous, seamless loop on a dedicated `"loading-audio"` LiveKit track, separate from the `"tts-audio"` speech track. +- If a loop is already running, the command is an idempotent no-op (the loop continues uninterrupted). +- The loading clip must have been supplied via the `loading_audio` object in the `config` message. See [Loading Indicator](#loading-indicator) for configuration and authoring guidance. + +**Response Behavior:** +- The loading start command is **fire-and-forget** — no response message is sent on success (mirrors `clear`). +- Failures are reported with an `error` message. + +**Errors** (emitted as `{"type": "error", "message": "..."}`): +- No `config` message has been received yet +- Audio is disabled (`audio: false`) +- No LiveKit room is configured for the session +- `loading_audio` was not configured, or failed to decode +- The loading track failed to publish + +**Loop Control:** +- The loop is controlled **exclusively** by `loading_start` and `loading_stop`. The `speak` and `clear` commands do **not** affect it — they act only on the speech (STT/TTS) pipeline. +- The loop also stops automatically on session teardown / disconnect (resource cleanup). + +**Use Cases:** +```javascript +// User finished speaking; application starts background work (LLM/tool calls) +{"type": "loading_start"} +// ... the human in the room hears the looping sound while the application is busy ... +``` + +--- + +#### 6. Loading Stop Message + +**Purpose:** Stop the loading indicator loop with a short fade-out. + +**Structure:** +```json +{ + "type": "loading_stop" +} +``` + +**Behavior:** +- Stops the loading loop with a short fade-out (~30 ms) so playback ends without an audible click. +- Idempotent and always silent: stopping when nothing is playing, or when there is no LiveKit room, is a harmless no-op and **never** an error (mirrors `clear`). + +**Response Behavior:** +- The loading stop command is **fire-and-forget** — no response message is sent. + +**Required Client Guidance:** +Because `speak` does **not** stop the loop, and the loading track and TTS track are independent and can be audible at the same time, applications **must send `loading_stop` before (or together with) `speak`** in the normal "thinking finished, now answer" flow. Otherwise the loading sound is heard underneath the spoken answer. + +**Use Cases:** +```javascript +// Application finished its background work; stop the loading sound, then answer +{"type": "loading_stop"} +{"type": "speak", "text": "Here is what I found..."} +``` + +--- + +#### 7. Send Message **Purpose:** Send custom messages through LiveKit data channel to other participants. @@ -474,7 +546,7 @@ If `allow_interruption: false` was set on the most recent speak command, the cle --- -#### 6. SIP Transfer Message +#### 8. SIP Transfer Message **Purpose:** Transfer an active SIP call to another phone number using SIP REFER. @@ -1437,6 +1509,105 @@ Use the token to connect to LiveKit room from web browsers or mobile apps using --- +## Loading Indicator + +The loading indicator is a short audio clip the server plays on a continuous, seamless loop into the LiveKit room while your application is busy ("thinking") and cannot yet answer. It is the audio equivalent of a spinner: it signals to the human participant that the agent is still working, instead of leaving the room silent. + +The loading audio plays on its **own dedicated, second published LiveKit audio track** (`"loading-audio"`), separate from the speech track (`"tts-audio"`), so it never interferes with text-to-speech output. + +The feature has three parts: +1. **Configuration** — the `config` message gains an optional `loading_audio` object. The server decodes and validates the clip once and holds it for the session. +2. **`loading_start`** — begins looping the clip into the room. +3. **`loading_stop`** — stops the loop with a short fade-out. + +The server only implements the mechanism. The *cadence* — when and how often the loading sound plays — is entirely your application's concern. + +### Loading Audio Configuration + +To use the loading indicator, include a `loading_audio` object in the `config` message: + +```json +{ + "type": "config", + "audio": true, + "stt_config": { ... }, + "tts_config": { ... }, + "livekit": { + "room_name": "conversation-room-123" + }, + "loading_audio": { + "data": "UklGRiQAAABXQVZFZm10IBAAAAABAAEA...", + "format": "wav", + "volume": 0.3 + } +} +``` + +**Fields:** + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `data` | string | Yes | Base64-encoded audio bytes — either a complete WAV file or raw PCM. | +| `format` | string | No | `"wav"` or `"pcm"`. If omitted, the server auto-detects: bytes beginning with the `RIFF`/`WAVE` signature are treated as WAV, otherwise as raw PCM. | +| `sample_rate` | integer | Conditional | Sample rate in Hz. **Required for raw PCM** (which has no header). Ignored for WAV (the header is authoritative). Accepted range: 8000–48000 Hz. | +| `channels` | integer | No | Channel count for raw PCM only: `1` (mono) or `2` (stereo). Defaults to `1`. Ignored for WAV. | +| `volume` | number | No | Playback volume of the loading audio, from `0.0` (silent) to `1.0` (the clip's authored level). Default `1.0`. Out-of-range values are clamped. Applied as amplitude scaling of the clip's samples at config time. | + +**Audio Requirements:** + +- **Only 16-bit PCM is supported** — a 16-bit integer WAV, or raw 16-bit little-endian PCM. Other bit depths and float formats are rejected. +- Decoded duration must be between ~250 ms and ~10 s (**1–4 s recommended**). +- Decoded size is capped (~1.9 MB). + +**WAV vs. raw PCM:** + +- **WAV** — supply a complete 16-bit integer WAV file. The header is authoritative, so `sample_rate` and `channels` in the config are ignored. +- **Raw PCM** — supply interleaved 16-bit little-endian PCM with no header. You **must** provide `sample_rate`; `channels` defaults to `1`. + +**Behavioral rules:** + +- `loading_audio` is processed only when `audio` is `true` **and** a `livekit` configuration is present in the same `config` message. If supplied without those, the server emits an `error` message explaining the requirement and continues the session **without** loading capability. +- A decode or validation failure of `loading_audio` is **non-fatal**: the server emits an `error` message and the session continues normally — only the loading feature is unavailable. STT, TTS, and LiveKit are unaffected. +- Existing clients that omit `loading_audio` are completely unaffected — the field is purely additive and optional. + +### Controlling the Loop + +The loop is controlled with the [`loading_start`](#5-loading-start-message) and [`loading_stop`](#6-loading-stop-message) commands, documented in the Incoming Messages section. + +- `loading_start` begins the loop (idempotent — a no-op if already looping). +- `loading_stop` stops the loop with a short fade-out (idempotent and always silent). +- The loop is controlled **exclusively** by these two commands. The `speak` and `clear` commands do **not** affect it — they act only on the speech (STT/TTS) pipeline. +- The loop also stops automatically on session teardown / disconnect. + +**Required client guidance:** Because `speak` does not stop the loop, and the loading track and TTS track are independent and can be audible at the same time, **send `loading_stop` before (or together with) `speak`** in the normal "thinking finished, now answer" flow. Otherwise the loading sound is heard underneath the spoken answer. + +### Authoring the Clip + +The server plays the clip **as supplied** (apart from the `volume` scaling and the stop fade-out): it does not resample, crossfade, or re-time it. Author the clip carefully: + +- **Short** — ideally 1–4 s. +- **Seamless** — the start and end amplitudes must match so the clip loops without an audible click. A clip whose start and end amplitudes differ may click at the loop point. +- **Quiet** — author the clip at a low level so it sits under, rather than competes with, speech. +- **16-bit PCM** — a 16-bit integer WAV or raw 16-bit little-endian PCM. + +Loudness can be controlled in two ways: by authoring the clip quietly, and by setting the `volume` field (which scales the clip's sample amplitudes at config time). A LiveKit publisher cannot adjust a track's output volume after the fact, so these are the only loudness controls available. + +### Separate Track Behavior + +The loading audio is published on its own LiveKit audio track (`"loading-audio"`), independent of the speech track (`"tts-audio"`). An audio-enabled session with `loading_audio` configured therefore publishes **two** audio tracks from the Sayna agent participant. See [LiveKit Integration](livekit_integration.md#published-audio-tracks) for details, including how the loading track appears in room-composite recordings. + +After a LiveKit publisher-timeout reconnect, an in-progress loop stops (the old audio source is gone). The client must re-send `loading_start` to resume the loop. + +### Typical Interaction Flow + +1. Open `/ws`. Send `config` with `audio: true`, an STT config, a TTS config, a `livekit` config, **and** a `loading_audio` object. Receive `ready`. +2. The user speaks; the server emits `stt_result` messages. The application starts its background work (LLM/tool calls). +3. The application sends `loading_start` — the human in the room hears the looping sound. +4. The application finishes its work, sends `loading_stop`, then sends `speak` with the answer text. Sending `loading_stop` first is the application's responsibility. +5. Repeat per turn. + +--- + ## Use Case Patterns Sayna's WebSocket API supports multiple integration patterns. Choose based on your application needs. diff --git a/src/docs/openapi.rs b/src/docs/openapi.rs index e971b58..8b47529 100644 --- a/src/docs/openapi.rs +++ b/src/docs/openapi.rs @@ -22,7 +22,9 @@ use crate::handlers::{ speak::SpeakRequest, voices::Voice, ws::{ - config::{LiveKitWebSocketConfig, STTWebSocketConfig, TTSWebSocketConfig}, + config::{ + LiveKitWebSocketConfig, LoadingAudioConfig, STTWebSocketConfig, TTSWebSocketConfig, + }, messages::{ IncomingMessage, OutgoingMessage, ParticipantConnectedInfo, ParticipantDisconnectedInfo, TrackSubscribedInfo, UnifiedMessage, @@ -106,6 +108,7 @@ use crate::handlers::{ STTWebSocketConfig, TTSWebSocketConfig, LiveKitWebSocketConfig, + LoadingAudioConfig, Pronunciation, )), modifiers(&SecurityAddon), diff --git a/src/handlers/ws/config.rs b/src/handlers/ws/config.rs index 1f3594f..da24a97 100644 --- a/src/handlers/ws/config.rs +++ b/src/handlers/ws/config.rs @@ -241,6 +241,42 @@ impl TTSWebSocketConfig { } } +/// Loading-indicator audio configuration supplied in the `config` message. +#[derive(Deserialize, Serialize, Clone)] +#[cfg_attr(feature = "openapi", derive(utoipa::ToSchema))] +pub struct LoadingAudioConfig { + /// Base64-encoded audio bytes — a complete WAV file or raw 16-bit PCM. + pub data: String, + /// Audio format: "wav" or "pcm". If omitted, the server auto-detects. + #[serde(skip_serializing_if = "Option::is_none")] + #[cfg_attr(feature = "openapi", schema(example = "wav"))] + pub format: Option, + /// Sample rate in Hz. Required for raw PCM; ignored for WAV (header is authoritative). + #[serde(skip_serializing_if = "Option::is_none")] + #[cfg_attr(feature = "openapi", schema(example = 16000))] + pub sample_rate: Option, + /// Channel count for raw PCM (1 = mono, 2 = stereo); defaults to 1. Ignored for WAV. + #[serde(skip_serializing_if = "Option::is_none")] + #[cfg_attr(feature = "openapi", schema(example = 1))] + pub channels: Option, + /// Playback volume from 0.0 (silent) to 1.0 (authored level). Default 1.0; out-of-range clamped. + #[serde(skip_serializing_if = "Option::is_none")] + #[cfg_attr(feature = "openapi", schema(example = 0.3))] + pub volume: Option, +} + +impl std::fmt::Debug for LoadingAudioConfig { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("LoadingAudioConfig") + .field("data", &format_args!("<{} base64 chars>", self.data.len())) + .field("format", &self.format) + .field("sample_rate", &self.sample_rate) + .field("channels", &self.channels) + .field("volume", &self.volume) + .finish() + } +} + /// Compute TTS configuration hash for caching. /// /// Includes a stable auth fingerprint so session-scoped credentials do not share diff --git a/src/handlers/ws/config_handler.rs b/src/handlers/ws/config_handler.rs index ccdfe28..c0f4799 100644 --- a/src/handlers/ws/config_handler.rs +++ b/src/handlers/ws/config_handler.rs @@ -18,14 +18,14 @@ use crate::{ tts::AudioData, voice_manager::{VoiceManager, VoiceManagerConfig}, }, - livekit::{LiveKitClient, LiveKitMediaMode}, + livekit::{LiveKitClient, LiveKitMediaMode, LoadingClip, decode_loading_clip}, state::AppState, }; use super::{ config::{ - LiveKitWebSocketConfig, STTWebSocketConfig, TTSWebSocketConfig, TurnDetectConfigUpdate, - VADConfigUpdate, compute_tts_config_hash, + LiveKitWebSocketConfig, LoadingAudioConfig, STTWebSocketConfig, TTSWebSocketConfig, + TurnDetectConfigUpdate, VADConfigUpdate, compute_tts_config_hash, }, messages::{ MessageRoute, OutgoingMessage, ParticipantConnectedInfo, ParticipantDisconnectedInfo, @@ -70,6 +70,7 @@ fn resolve_stream_id(stream_id: Option) -> String { /// * `stt_ws_config` - STT provider configuration /// * `tts_ws_config` - TTS provider configuration /// * `livekit_ws_config` - Optional LiveKit configuration +/// * `loading_audio` - Optional loading-indicator audio configuration /// * `state` - Connection state to update /// * `message_tx` - Channel for sending response messages /// * `app_state` - Application state containing API keys @@ -83,6 +84,7 @@ pub async fn handle_config_message( stt_ws_config: Option, tts_ws_config: Option, livekit_ws_config: Option, + loading_audio: Option, state: &Arc>, message_tx: &mpsc::Sender, app_state: &Arc, @@ -113,6 +115,40 @@ pub async fn handle_config_message( } debug!(stream_id = %stream_id, "Stored stream_id in connection state"); + // Decode the optional loading-indicator audio clip (non-fatal on failure). + let loading_clip = match loading_audio { + None => None, + Some(cfg) => { + if !audio_enabled || livekit_ws_config.is_none() { + let _ = message_tx + .send(MessageRoute::Outgoing(OutgoingMessage::Error { + message: + "loading_audio requires audio to be enabled and a livekit configuration" + .to_string(), + })) + .await; + None + } else { + match decode_loading_clip( + &cfg.data, + cfg.format.as_deref(), + cfg.sample_rate, + cfg.channels, + cfg.volume, + ) { + Ok(clip) => Some(clip), + Err(message) => { + error!("Failed to decode loading_audio: {}", message); + let _ = message_tx + .send(MessageRoute::Outgoing(OutgoingMessage::Error { message })) + .await; + None + } + } + } + } + }; + // Initialize voice manager if audio is enabled let voice_manager = if audio_enabled { match initialize_voice_manager( @@ -161,6 +197,7 @@ pub async fn handle_config_message( livekit_ws_config, audio_enabled, tts_ws_config.as_ref(), + loading_clip, &app_state.config.livekit_url, voice_manager.as_ref(), message_tx, @@ -703,6 +740,8 @@ async fn register_final_tts_callback( /// Initialize LiveKit client and set up callbacks /// /// # Arguments +/// * `loading_clip` - Optional decoded loading-indicator audio clip. When present, +/// it is registered on the client before connecting so `loading_start` can use it. /// * `auth_id` - Optional tenant identifier. If provided, enforced via room metadata /// to prevent cross-tenant room access. #[allow(clippy::too_many_arguments)] @@ -710,6 +749,7 @@ async fn initialize_livekit_client( livekit_ws_config: LiveKitWebSocketConfig, audio_enabled: bool, tts_config: Option<&TTSWebSocketConfig>, + loading_clip: Option, livekit_url: &str, voice_manager: Option<&Arc>, message_tx: &mpsc::Sender, @@ -899,6 +939,11 @@ async fn initialize_livekit_client( livekit_ws_config.to_livekit_config(agent_token, audio_enabled, tts_config, livekit_url); let mut livekit_client = LiveKitClient::new(livekit_config); + // Register the loading-indicator audio clip before connecting, if supplied. + if let Some(clip) = loading_clip { + livekit_client.set_loading_audio_clip(clip); + } + // Set up audio callback to forward to STT processing if audio_enabled && let Some(vm) = voice_manager { setup_livekit_audio_callback(&mut livekit_client, vm, message_tx); diff --git a/src/handlers/ws/loading_handler.rs b/src/handlers/ws/loading_handler.rs new file mode 100644 index 0000000..68d2de1 --- /dev/null +++ b/src/handlers/ws/loading_handler.rs @@ -0,0 +1,191 @@ +//! Loading-indicator audio handler for WebSocket connections +//! +//! This module handles the `loading_start` and `loading_stop` control messages. +//! The loading-indicator audio loop is controlled exclusively by these two +//! commands — the `speak` and `clear` commands never start, stop, or otherwise +//! affect it. + +use std::sync::Arc; +use tokio::sync::{RwLock, mpsc}; +use tracing::{debug, error}; + +use crate::AppError; + +use super::{ + messages::{MessageRoute, OutgoingMessage}, + state::ConnectionState, +}; + +/// Send an error message back to the WebSocket client. +/// +/// Send failures are ignored because they only mean the client has already +/// disconnected. +async fn send_error(message_tx: &mpsc::Sender, message: impl Into) { + let _ = message_tx + .send(MessageRoute::Outgoing(OutgoingMessage::Error { + message: message.into(), + })) + .await; +} + +/// Handle the `loading_start` command. +/// +/// Begins looping the configured loading-indicator audio into the LiveKit room. +/// Requires that audio is enabled and a LiveKit room is configured; otherwise an +/// error message is sent back to the client. +/// +/// # Arguments +/// * `state` - Connection state containing the LiveKit client +/// * `message_tx` - Channel for sending response messages back to the client +/// +/// # Returns +/// * `bool` - always `true`; this handler never terminates the connection +pub async fn handle_loading_start_message( + state: &Arc>, + message_tx: &mpsc::Sender, +) -> bool { + debug!("Processing loading_start command"); + + // Snapshot the state we need, then drop the read lock immediately. + let (audio_enabled, livekit_client) = { + let state_guard = state.read().await; + ( + state_guard.is_audio_enabled(), + state_guard.livekit_client.clone(), + ) + }; + + // Audio must be enabled. This also covers the "no config received yet" case, + // since audio defaults to disabled. + if !audio_enabled { + send_error( + message_tx, + "Loading indicator requires audio to be enabled. Send a config message with audio=true first.", + ) + .await; + return true; + } + + // A LiveKit room is required to play the loading audio. + let livekit_client = match livekit_client { + Some(client) => client, + None => { + send_error( + message_tx, + "Loading indicator requires a LiveKit room. Include a livekit configuration in your config message.", + ) + .await; + return true; + } + }; + + // `start_loading_audio` takes `&self`, so a read lock is sufficient. + let client_guard = livekit_client.read().await; + if let Err(e) = client_guard.start_loading_audio().await { + let message = match e { + AppError::BadRequest(m) + | AppError::InternalServerError(m) + | AppError::NotFound(m) + | AppError::Unauthorized(m) => m, + }; + error!("Failed to start loading-indicator audio: {}", message); + send_error(message_tx, message).await; + } else { + debug!("Loading-indicator audio started"); + } + + true +} + +/// Handle the `loading_stop` command. +/// +/// Stops the loading-indicator audio loop (with a short fade-out). When there is +/// no LiveKit client this is a silent no-op: no message and no error is sent. +/// +/// # Arguments +/// * `state` - Connection state containing the LiveKit client +/// +/// # Returns +/// * `bool` - always `true`; this handler never terminates the connection +pub async fn handle_loading_stop_message(state: &Arc>) -> bool { + debug!("Processing loading_stop command"); + + // Snapshot the LiveKit client, then drop the read lock immediately. + let livekit_client = { + let state_guard = state.read().await; + state_guard.livekit_client.clone() + }; + + // Silent no-op when there is no LiveKit client. + if let Some(client) = livekit_client { + // `stop_loading_audio` takes `&self`, so a read lock is sufficient. + let client_guard = client.read().await; + client_guard.stop_loading_audio().await; + debug!("Loading-indicator audio stopped"); + } else { + debug!("No LiveKit client - loading_stop is a no-op"); + } + + true +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn test_handle_loading_start_message_rejects_when_audio_disabled() { + let state = Arc::new(RwLock::new(ConnectionState::new())); + let (message_tx, mut message_rx) = mpsc::channel(4); + + let continue_processing = handle_loading_start_message(&state, &message_tx).await; + + assert!(continue_processing); + match message_rx.recv().await { + Some(MessageRoute::Outgoing(OutgoingMessage::Error { message })) => { + assert!(message.contains("audio to be enabled")); + } + Some(_) => panic!("expected audio-disabled error"), + None => panic!("expected audio-disabled error"), + } + } + + #[tokio::test] + async fn test_handle_loading_start_message_rejects_when_no_livekit() { + let state = Arc::new(RwLock::new(ConnectionState::new())); + { + let state_guard = state.write().await; + state_guard.set_audio_enabled(true); + } + let (message_tx, mut message_rx) = mpsc::channel(4); + + let continue_processing = handle_loading_start_message(&state, &message_tx).await; + + assert!(continue_processing); + match message_rx.recv().await { + Some(MessageRoute::Outgoing(OutgoingMessage::Error { message })) => { + assert!(message.contains("LiveKit room")); + } + Some(_) => panic!("expected missing-LiveKit error"), + None => panic!("expected missing-LiveKit error"), + } + } + + #[tokio::test] + async fn test_handle_loading_stop_message_is_silent_noop() { + let state = Arc::new(RwLock::new(ConnectionState::new())); + let (message_tx, mut message_rx): ( + mpsc::Sender, + mpsc::Receiver, + ) = mpsc::channel(4); + + let continue_processing = handle_loading_stop_message(&state).await; + drop(message_tx); + + assert!(continue_processing); + assert!( + message_rx.try_recv().is_err(), + "loading_stop must not emit any message when there is no LiveKit client" + ); + } +} diff --git a/src/handlers/ws/messages.rs b/src/handlers/ws/messages.rs index fd068a0..6eea30f 100644 --- a/src/handlers/ws/messages.rs +++ b/src/handlers/ws/messages.rs @@ -8,8 +8,8 @@ use bytes::Bytes; use serde::{Deserialize, Serialize}; use super::config::{ - LiveKitWebSocketConfig, STTWebSocketConfig, TTSWebSocketConfig, TurnDetectConfigUpdate, - VADConfigUpdate, default_allow_interruption, default_audio_enabled, + LiveKitWebSocketConfig, LoadingAudioConfig, STTWebSocketConfig, TTSWebSocketConfig, + TurnDetectConfigUpdate, VADConfigUpdate, default_allow_interruption, default_audio_enabled, }; /// WebSocket message types for incoming messages @@ -44,6 +44,9 @@ pub enum IncomingMessage { /// Optional LiveKit configuration for real-time audio streaming #[serde(skip_serializing_if = "Option::is_none")] livekit: Option, + /// Optional loading-indicator audio configuration. + #[serde(skip_serializing_if = "Option::is_none")] + loading_audio: Option, }, #[serde(rename = "speak")] Speak { @@ -109,6 +112,12 @@ pub enum IncomingMessage { #[serde(skip_serializing_if = "Option::is_none")] turn_detect: Option, }, + /// Begin looping the configured loading-indicator audio into the LiveKit room. + #[serde(rename = "loading_start")] + LoadingStart, + /// Stop the loading-indicator audio loop (with a short fade-out). + #[serde(rename = "loading_stop")] + LoadingStop, } /// Unified message structure for all incoming messages from various sources @@ -471,6 +480,108 @@ mod tests { assert!(json.contains(r#""message":"Invalid phone number""#)); } + #[test] + fn test_loading_start_message_deserialization() { + let json = r#"{"type":"loading_start"}"#; + let msg: IncomingMessage = serde_json::from_str(json).expect("Should deserialize"); + + match msg { + IncomingMessage::LoadingStart => {} + _ => panic!("Expected LoadingStart variant"), + } + } + + #[test] + fn test_loading_start_message_serialization() { + let msg = IncomingMessage::LoadingStart; + + let json = serde_json::to_string(&msg).expect("Should serialize"); + assert!(json.contains(r#""type":"loading_start""#)); + } + + #[test] + fn test_loading_stop_message_deserialization() { + let json = r#"{"type":"loading_stop"}"#; + let msg: IncomingMessage = serde_json::from_str(json).expect("Should deserialize"); + + match msg { + IncomingMessage::LoadingStop => {} + _ => panic!("Expected LoadingStop variant"), + } + } + + #[test] + fn test_loading_stop_message_serialization() { + let msg = IncomingMessage::LoadingStop; + + let json = serde_json::to_string(&msg).expect("Should serialize"); + assert!(json.contains(r#""type":"loading_stop""#)); + } + + #[test] + fn test_loading_audio_config_round_trip() { + let cfg = LoadingAudioConfig { + data: "QUJDRA==".to_string(), + format: Some("wav".to_string()), + sample_rate: Some(16000), + channels: Some(1), + volume: Some(0.3), + }; + + let json = serde_json::to_string(&cfg).expect("Should serialize"); + let parsed: LoadingAudioConfig = serde_json::from_str(&json).expect("Should deserialize"); + + assert_eq!(parsed.data, "QUJDRA=="); + assert_eq!(parsed.format, Some("wav".to_string())); + assert_eq!(parsed.sample_rate, Some(16000)); + assert_eq!(parsed.channels, Some(1)); + assert_eq!(parsed.volume, Some(0.3)); + } + + #[test] + fn test_loading_audio_config_round_trip_minimal() { + let cfg = LoadingAudioConfig { + data: "QUJDRA==".to_string(), + format: None, + sample_rate: None, + channels: None, + volume: None, + }; + + let json = serde_json::to_string(&cfg).expect("Should serialize"); + // Optional fields are omitted when None. + assert!(!json.contains("format")); + assert!(!json.contains("sample_rate")); + assert!(!json.contains("channels")); + assert!(!json.contains("volume")); + + let parsed: LoadingAudioConfig = serde_json::from_str(&json).expect("Should deserialize"); + assert_eq!(parsed.data, "QUJDRA=="); + assert!(parsed.format.is_none()); + assert!(parsed.sample_rate.is_none()); + assert!(parsed.channels.is_none()); + assert!(parsed.volume.is_none()); + } + + #[test] + fn test_loading_audio_config_debug_redacts_data() { + let cfg = LoadingAudioConfig { + data: "SECRET_BASE64_PAYLOAD_VALUE".to_string(), + format: Some("pcm".to_string()), + sample_rate: Some(16000), + channels: Some(1), + volume: Some(1.0), + }; + + let debug = format!("{cfg:?}"); + // The redacting Debug reports the length, never the raw data. + assert!(debug.contains("base64 chars")); + assert!( + !debug.contains("SECRET_BASE64_PAYLOAD_VALUE"), + "Debug output must not leak the raw base64 data" + ); + } + #[test] fn test_vad_event_serialization() { let event = OutgoingMessage::VadEvent { diff --git a/src/handlers/ws/mod.rs b/src/handlers/ws/mod.rs index 65a6b3c..cc326f2 100644 --- a/src/handlers/ws/mod.rs +++ b/src/handlers/ws/mod.rs @@ -19,6 +19,8 @@ //! - `{"type": "config", "audio": true, "stt_config": {...}, "tts_config": {...}, "livekit": {...}}` - Initialize voice providers and optionally connect to LiveKit //! - `{"type": "speak", "text": "Hello world", "flush": true, "allow_interruption": true}` - Synthesize speech from text (flush and allow_interruption are optional, both default to true) //! - `{"type": "clear"}` - Clear pending TTS audio and clear queue (ignored if allow_interruption=false until audio finishes) +//! - `{"type": "loading_start"}` - Begin looping the configured loading-indicator audio into the LiveKit room (requires audio enabled and a LiveKit room) +//! - `{"type": "loading_stop"}` - Stop the loading-indicator audio loop (no-op when no LiveKit room is configured) //! - `{"type": "send_message", "message": "Hello LiveKit!", "role": "user", "topic": "chat"}` - Send custom text message through LiveKit (topic is optional) //! - `{"type": "sip_transfer", "transfer_to": "+1234567890"}` - Transfer active SIP call to another phone number //! - **Binary messages** - Raw audio data for transcription @@ -435,6 +437,7 @@ pub mod config; pub mod config_handler; pub mod error; pub mod handler; +pub mod loading_handler; pub mod messages; pub mod processor; pub mod state; diff --git a/src/handlers/ws/processor.rs b/src/handlers/ws/processor.rs index 3cdb1ea..de3c0a3 100644 --- a/src/handlers/ws/processor.rs +++ b/src/handlers/ws/processor.rs @@ -12,6 +12,7 @@ use super::{ audio_handler::{handle_clear_message, handle_speak_message}, command_handler::{handle_send_message, handle_sip_transfer}, config_handler::{handle_config_message, handle_update_config_message}, + loading_handler::{handle_loading_start_message, handle_loading_stop_message}, messages::{IncomingMessage, MessageRoute}, state::ConnectionState, }; @@ -48,9 +49,18 @@ pub async fn handle_incoming_message( stt_config, tts_config, livekit, + loading_audio, } => { handle_config_message( - stream_id, audio, stt_config, tts_config, livekit, state, message_tx, app_state, + stream_id, + audio, + stt_config, + tts_config, + livekit, + loading_audio, + state, + message_tx, + app_state, ) .await } @@ -72,5 +82,7 @@ pub async fn handle_incoming_message( IncomingMessage::UpdateConfig { vad, turn_detect } => { handle_update_config_message(vad, turn_detect, message_tx, app_state).await } + IncomingMessage::LoadingStart => handle_loading_start_message(state, message_tx).await, + IncomingMessage::LoadingStop => handle_loading_stop_message(state).await, } } diff --git a/src/handlers/ws/tests.rs b/src/handlers/ws/tests.rs index 62c88b9..5d19092 100644 --- a/src/handlers/ws/tests.rs +++ b/src/handlers/ws/tests.rs @@ -3,7 +3,8 @@ use bytes::Bytes; use serde_json::json; use super::config::{ - LiveKitWebSocketConfig, STTWebSocketConfig, TTSWebSocketConfig, compute_tts_config_hash, + LiveKitWebSocketConfig, LoadingAudioConfig, STTWebSocketConfig, TTSWebSocketConfig, + compute_tts_config_hash, }; use super::messages::{ IncomingMessage, MessageRoute, OutgoingMessage, ParticipantDisconnectedInfo, UnifiedMessage, @@ -100,6 +101,7 @@ fn test_incoming_message_serialization() { auth: None, }), livekit: None, + loading_audio: None, }; let json = serde_json::to_string(&config_msg).unwrap(); @@ -865,6 +867,7 @@ fn test_incoming_message_config_with_livekit() { sayna_participant_name: None, listen_participants: vec![], }), + loading_audio: None, }; let json = serde_json::to_string(&config_msg).unwrap(); @@ -901,6 +904,7 @@ fn test_incoming_message_config_without_livekit() { auth: None, }), livekit: None, + loading_audio: None, }; let json = serde_json::to_string(&config_msg).unwrap(); @@ -1201,6 +1205,7 @@ fn test_config_message_without_livekit_routing() { auth: None, }), livekit: None, // No LiveKit configuration + loading_audio: None, }; let json = serde_json::to_string(&config_msg).unwrap(); @@ -1254,6 +1259,7 @@ fn test_config_message_with_livekit_routing() { sayna_participant_name: None, listen_participants: vec![], }), + loading_audio: None, }; let json = serde_json::to_string(&config_msg).unwrap(); @@ -1441,6 +1447,7 @@ fn test_config_message_audio_disabled() { sayna_participant_name: None, listen_participants: vec![], }), + loading_audio: None, }; let json = serde_json::to_string(&config_msg).unwrap(); @@ -1501,6 +1508,7 @@ fn test_config_message_audio_default() { auth: None, }), livekit: None, + loading_audio: None, }; let json = serde_json::to_string(&config_msg).unwrap(); @@ -1574,3 +1582,134 @@ fn test_unified_message_for_livekit_integration() { // Should not contain message field for binary data assert!(!json_binary.contains("\"message\":null")); } + +#[test] +fn test_parse_config_message_with_loading_audio() { + let json = r#"{ + "type": "config", + "stt_config": { + "provider": "deepgram", + "language": "en-US", + "sample_rate": 16000, + "channels": 1, + "punctuation": true, + "encoding": "linear16", + "model": "nova-3" + }, + "tts_config": { + "provider": "deepgram", + "voice_id": "aura-luna-en", + "speaking_rate": 1.0, + "audio_format": "pcm", + "sample_rate": 22050, + "connection_timeout": 30, + "request_timeout": 60, + "model": "" + }, + "loading_audio": { + "data": "QUJDRA==", + "format": "wav", + "sample_rate": 16000, + "channels": 1, + "volume": 0.3 + } + }"#; + + let parsed: IncomingMessage = serde_json::from_str(json).unwrap(); + if let IncomingMessage::Config { loading_audio, .. } = parsed { + let loading_audio = loading_audio.expect("loading_audio should be populated"); + assert_eq!(loading_audio.data, "QUJDRA=="); + assert_eq!(loading_audio.format, Some("wav".to_string())); + assert_eq!(loading_audio.sample_rate, Some(16000)); + } else { + panic!("Expected Config message"); + } +} + +#[test] +fn test_parse_config_message_without_loading_audio() { + // Existing clients omit loading_audio entirely; it must default to None. + let json = r#"{ + "type": "config", + "stt_config": { + "provider": "deepgram", + "language": "en-US", + "sample_rate": 16000, + "channels": 1, + "punctuation": true, + "encoding": "linear16", + "model": "nova-3" + }, + "tts_config": { + "provider": "deepgram", + "voice_id": "aura-luna-en", + "speaking_rate": 1.0, + "audio_format": "pcm", + "sample_rate": 22050, + "connection_timeout": 30, + "request_timeout": 60, + "model": "" + } + }"#; + + let parsed: IncomingMessage = serde_json::from_str(json).unwrap(); + if let IncomingMessage::Config { loading_audio, .. } = parsed { + assert!( + loading_audio.is_none(), + "loading_audio should be None when not provided" + ); + } else { + panic!("Expected Config message"); + } +} + +#[test] +fn test_config_message_loading_audio_round_trip() { + let config_msg = IncomingMessage::Config { + stream_id: None, + audio: Some(true), + stt_config: Some(STTWebSocketConfig { + provider: "deepgram".to_string(), + language: "en-US".to_string(), + sample_rate: 16000, + channels: 1, + punctuation: true, + encoding: "linear16".to_string(), + model: "nova-3".to_string(), + auth: None, + }), + tts_config: Some(TTSWebSocketConfig { + provider: "deepgram".to_string(), + voice_id: Some("aura-luna-en".to_string()), + speaking_rate: Some(1.0), + audio_format: Some("pcm".to_string()), + sample_rate: Some(22050), + connection_timeout: Some(30), + request_timeout: Some(60), + model: "".to_string(), + pronunciations: Vec::new(), + auth: None, + }), + livekit: None, + loading_audio: Some(LoadingAudioConfig { + data: "QUJDRA==".to_string(), + format: Some("pcm".to_string()), + sample_rate: Some(16000), + channels: Some(1), + volume: Some(0.5), + }), + }; + + let json = serde_json::to_string(&config_msg).unwrap(); + let parsed: IncomingMessage = serde_json::from_str(&json).unwrap(); + if let IncomingMessage::Config { loading_audio, .. } = parsed { + let loading_audio = loading_audio.expect("loading_audio should survive round-trip"); + assert_eq!(loading_audio.data, "QUJDRA=="); + assert_eq!(loading_audio.format, Some("pcm".to_string())); + assert_eq!(loading_audio.sample_rate, Some(16000)); + assert_eq!(loading_audio.channels, Some(1)); + assert_eq!(loading_audio.volume, Some(0.5)); + } else { + panic!("Expected Config message"); + } +} diff --git a/src/livekit/client/audio.rs b/src/livekit/client/audio.rs index b8f7ed6..93cf73e 100644 --- a/src/livekit/client/audio.rs +++ b/src/livekit/client/audio.rs @@ -606,6 +606,43 @@ impl LiveKitClient { info!("Successfully reconnected and re-published audio track"); + // Stop any in-progress loading-audio loop: its audio source died with the + // pre-reconnect room. A buffered `NativeAudioSource` keeps accepting frames + // even after its room is gone, so the loop cannot detect the reconnect on + // its own — cancel and abort it here. Clearing `loading_loop` lets a later + // `loading_start` spawn a fresh loop on the re-published track; the client + // must re-send `loading_start` to resume the loading sound. + if let Some(loading) = ctx.loading_loop.lock().await.take() { + loading.token.cancel(); + loading.handle.abort(); + } + + // Re-publish the loading-audio track if a loading clip is configured + // (non-fatal). The re-published `LocalAudioTrack` handle is intentionally + // dropped here: `OperationContext` carries no `loading_audio_track` field, + // and the publication alone keeps the track alive — mirroring how the TTS + // track is re-published above. + if let Some(clip) = &ctx.loading_clip { + *ctx.loading_audio_source.lock().await = None; + *ctx.loading_track_publication.lock().await = None; + match super::loading_audio::publish_loading_audio_track( + &local_participant, + clip.sample_rate(), + clip.channels(), + ) + .await + { + Ok((source, _track, publication)) => { + *ctx.loading_audio_source.lock().await = Some(source); + *ctx.loading_track_publication.lock().await = Some(publication); + info!("Re-published loading audio track after reconnect"); + } + Err(e) => warn!( + "Failed to re-publish loading audio track after reconnect (non-fatal): {e:?}" + ), + } + } + // Return room_events for event handler restart and success flag Ok((Some(room_events), true)) } diff --git a/src/livekit/client/connection.rs b/src/livekit/client/connection.rs index 29bec70..63f22eb 100644 --- a/src/livekit/client/connection.rs +++ b/src/livekit/client/connection.rs @@ -92,6 +92,19 @@ impl LiveKitClient { let _ = tokio::time::timeout(std::time::Duration::from_millis(500), handle).await; } + // Tear down the loading-audio loop: cancel, await with timeout, abort as backstop. + if let Some(loading) = self.loading_loop.lock().await.take() { + loading.token.cancel(); + let abort = loading.handle.abort_handle(); + if tokio::time::timeout(std::time::Duration::from_millis(500), loading.handle) + .await + .is_err() + { + abort.abort(); + warn!("Loading audio loop did not stop within timeout; aborted"); + } + } + // Clear the operation queue self.operation_queue = None; @@ -101,6 +114,13 @@ impl LiveKitClient { *self.local_track_publication.lock().await = None; self.has_audio_source_atomic.store(false, Ordering::Release); + // Clear loading-audio resources only after the loop has been awaited, + // so no stray fade-out frame is captured into a half-torn-down source. + *self.loading_audio_source.lock().await = None; + self.loading_audio_track = None; + *self.loading_track_publication.lock().await = None; + self.loading_clip = None; + *self.room.lock().await = None; self.room_events = None; @@ -134,6 +154,9 @@ impl LiveKitClient { if self.config.publish_audio { self.setup_audio_publishing().await?; + if let Err(e) = self.setup_loading_audio_track().await { + warn!("Loading audio track setup failed (non-fatal): {e:?}"); + } } else { info!("LiveKit connected in strict no-media mode; skipping audio publishing"); } diff --git a/src/livekit/client/loading_audio.rs b/src/livekit/client/loading_audio.rs new file mode 100644 index 0000000..25c6256 --- /dev/null +++ b/src/livekit/client/loading_audio.rs @@ -0,0 +1,562 @@ +//! Dedicated "loading-audio" track and the loading-audio playback loop. +//! +//! This submodule owns a *second*, parallel published LiveKit audio track — +//! named [`LOADING_AUDIO_TRACK_NAME`] — that is fully independent of the TTS +//! track (`"tts-audio"`). It publishes that track, and runs a background loop +//! that feeds the configured [`LoadingClip`] into it on a seamless cursor-based +//! loop until cancelled, finishing with a short linear fade-out. +//! +//! ## Isolation invariant +//! +//! The loading-audio loop NEVER touches the TTS path's `operation_queue`, +//! `audio_queue`, `audio_generation`, or `has_audio_source_atomic`. It captures +//! frames directly on its own dedicated `loading_audio_source`. The loading +//! track is a second, parallel published track, independent of the TTS track. + +use std::sync::Arc; + +use livekit::options::TrackPublishOptions; +use livekit::prelude::LocalTrackPublication; +use livekit::track::{LocalAudioTrack, LocalTrack, TrackSource}; +use livekit::webrtc::audio_source::native::NativeAudioSource; +use livekit::webrtc::prelude::{AudioFrame, AudioSourceOptions, RtcAudioSource}; +use tokio_util::sync::CancellationToken; +use tracing::{debug, info, warn}; + +use super::LiveKitClient; +use crate::AppError; +use crate::livekit::loading_clip::LoadingClip; + +/// Name of the published LiveKit audio track that carries the loading clip. +pub(super) const LOADING_AUDIO_TRACK_NAME: &str = "loading-audio"; + +/// Length of the linear fade-out applied when the loop stops, in milliseconds. +/// +/// Roughly three 10 ms frames; long enough to avoid an audible click without a +/// perceptible tail. +const LOADING_AUDIO_FADE_OUT_MS: u32 = 30; + +/// Depth of the loading audio source's internal buffering queue, in milliseconds. +/// +/// `NativeAudioSource::new` interprets its fourth argument as a queue size in +/// milliseconds and asserts that it is a non-zero multiple of 10. A fixed value +/// — independent of the clip's (client-supplied) sample rate — guarantees that +/// an arbitrary sample rate can never produce an invalid queue size. 100 ms +/// mirrors LiveKit's `BackgroundAudioPlayer` reference design: once the queue +/// fills, `capture_frame().await` applies real-time back-pressure, pacing the +/// loop at one 10 ms frame per 10 ms of wall-clock with no manual timer. +pub(super) const LOADING_AUDIO_QUEUE_SIZE_MS: u32 = 100; + +/// Handle to a running loading-audio loop: its task handle and cancellation token. +/// +/// Visibility is `pub(crate)` so it can name the `pub(super)` `loading_loop` +/// field on the `pub` [`LiveKitClient`] struct without a private-interface +/// warning; it is only ever constructed and consumed within the `client` +/// submodule. +pub(crate) struct LoadingLoopHandle { + /// Join handle of the spawned loop task. + pub(super) handle: tokio::task::JoinHandle<()>, + /// Token used to request that the loop fade out and exit. + pub(super) token: CancellationToken, +} + +/// Returns the interleaved `i16` sample count contained in one 10 ms frame. +/// +/// For mono this equals `sample_rate / 100`; for stereo it is twice that. +pub(super) fn loading_frame_len(sample_rate: u32, channels: u16) -> usize { + (sample_rate as usize / 100) * channels as usize +} + +/// Fills `out` with the next `out.len()` interleaved samples taken from +/// `clip_samples`, wrapping seamlessly at the end of the clip. +/// +/// `cursor` is the current read position into `clip_samples`. The new cursor +/// (modulo the clip length) is returned. An empty `clip_samples` is handled +/// safely: `out` is zero-filled and `0` is returned, with no panic. +pub(super) fn fill_loading_frame(clip_samples: &[i16], cursor: usize, out: &mut [i16]) -> usize { + if clip_samples.is_empty() { + out.fill(0); + return 0; + } + + let len = clip_samples.len(); + let mut pos = cursor % len; + for slot in out.iter_mut() { + *slot = clip_samples[pos]; + pos += 1; + if pos == len { + pos = 0; + } + } + pos +} + +/// Returns the linear fade-out factor for fade sample `index` of `total` total +/// fade samples, ramping from near `1.0` down to exactly `0.0`. +/// +/// `fade_factor(total - 1, total)` is `0.0`, the value is monotonically +/// non-increasing in `index`, and `total == 0` yields `0.0` safely. +pub(super) fn fade_factor(index: usize, total: usize) -> f32 { + if total == 0 { + 0.0 + } else { + (1.0 - (index + 1) as f32 / total as f32).max(0.0) + } +} + +/// Applies the linear fade-out ramp to `frame` in place, beginning at fade +/// sample `start_index` of `total` total fade samples. Returns the fade index +/// one past the last sample scaled, so a following frame continues the ramp. +/// +/// Each sample is scaled by [`fade_factor`] and rounded to the nearest `i16`. +/// Because the factor is within `0.0..=1.0`, the scaled magnitude never exceeds +/// the input magnitude, so the `as i16` cast cannot overflow or wrap. +pub(super) fn apply_fade(frame: &mut [i16], start_index: usize, total: usize) -> usize { + let mut index = start_index; + for slot in frame.iter_mut() { + let factor = fade_factor(index, total); + *slot = (f32::from(*slot) * factor).round() as i16; + index += 1; + } + index +} + +/// Creates the dedicated loading audio source plus the "loading-audio" track +/// and publishes it on the given participant. +/// +/// On success returns the audio source, the track handle (which must be kept +/// alive to keep the track published), and the resulting publication. +pub(super) async fn publish_loading_audio_track( + local_participant: &livekit::prelude::LocalParticipant, + sample_rate: u32, + channels: u16, +) -> Result< + ( + Arc, + LocalAudioTrack, + LocalTrackPublication, + ), + AppError, +> { + // `auto_gain_control: false` is load-bearing for the `volume` feature: + // WebRTC AGC would re-normalize loudness and silently undo the configured + // volume attenuation already applied to the clip's samples. + let audio_source_options = AudioSourceOptions { + echo_cancellation: false, + noise_suppression: false, + auto_gain_control: false, + }; + + let source = Arc::new(NativeAudioSource::new( + audio_source_options, + sample_rate, + channels as u32, + LOADING_AUDIO_QUEUE_SIZE_MS, + )); + + let rtc_audio_source = RtcAudioSource::Native((*source).clone()); + let track = LocalAudioTrack::create_audio_track(LOADING_AUDIO_TRACK_NAME, rtc_audio_source); + + let publish_options = TrackPublishOptions { + source: TrackSource::Microphone, + ..Default::default() + }; + + match local_participant + .publish_track(LocalTrack::Audio(track.clone()), publish_options) + .await + { + Ok(publication) => Ok((source, track, publication)), + Err(e) => Err(AppError::InternalServerError(format!( + "Failed to publish loading audio track: {e:?}" + ))), + } +} + +impl LiveKitClient { + /// Stores the loading audio clip to play while the agent is busy. + /// + /// Must be called before [`LiveKitClient::connect`]; the clip is published + /// as a dedicated track during connection bootstrap. + pub fn set_loading_audio_clip(&mut self, clip: LoadingClip) { + self.loading_clip = Some(Arc::new(clip)); + } + + /// Publishes the dedicated "loading-audio" track if a loading clip is set. + /// + /// When no loading clip is configured this is a clean no-op (the feature is + /// simply unused for this session). On a publish failure the error is + /// propagated; the caller treats loading-track setup as non-fatal. + pub(super) async fn setup_loading_audio_track(&mut self) -> Result<(), AppError> { + let clip = match &self.loading_clip { + Some(clip) => Arc::clone(clip), + None => return Ok(()), + }; + + // Extract local participant while holding the room lock briefly. + let local_participant = { + let room_guard = self.room.lock().await; + if let Some(room) = &*room_guard { + room.local_participant().clone() + } else { + return Err(AppError::InternalServerError( + "Room not available for loading audio track publishing".to_string(), + )); + } + }; + + let (source, track, publication) = + publish_loading_audio_track(&local_participant, clip.sample_rate(), clip.channels()) + .await?; + + *self.loading_audio_source.lock().await = Some(source); + self.loading_audio_track = Some(track); + *self.loading_track_publication.lock().await = Some(publication); + + info!( + "Successfully published loading audio track: {}Hz, {} channels", + clip.sample_rate(), + clip.channels() + ); + Ok(()) + } + + /// Starts looping the configured loading clip into the "loading-audio" track. + /// + /// Requires an active LiveKit connection and a configured, published loading + /// clip. Calling this while a loop is already running is an idempotent + /// no-op — including during the brief fade-out after a `stop_loading_audio`, + /// which still counts as running. The loop runs until + /// [`LiveKitClient::stop_loading_audio`] (or disconnect) cancels it. + /// + /// # Errors + /// + /// Returns an error if the client is not connected, or if no loading audio + /// clip / source is available for this session. + pub async fn start_loading_audio(&self) -> Result<(), AppError> { + if !self.is_connected() { + return Err(AppError::InternalServerError( + "Not connected to LiveKit room".to_string(), + )); + } + + let source = { + let guard = self.loading_audio_source.lock().await; + guard.as_ref().map(Arc::clone) + }; + let source = match source { + Some(source) => source, + None => { + return Err(AppError::BadRequest( + "loading audio is not available for this session".to_string(), + )); + } + }; + + let clip = match &self.loading_clip { + Some(clip) => Arc::clone(clip), + None => { + return Err(AppError::BadRequest( + "loading audio is not available for this session".to_string(), + )); + } + }; + + let mut loop_guard = self.loading_loop.lock().await; + // A finished handle is stale and is replaced; a live one means the loop + // is already running, so starting again is an idempotent no-op. + if let Some(existing) = loop_guard.as_ref() + && !existing.handle.is_finished() + { + debug!("Loading audio loop already running; start request ignored"); + return Ok(()); + } + + let token = CancellationToken::new(); + // Isolation invariant: the spawned loop owns only its dedicated audio + // source, the loading clip, and its cancellation token. It never + // touches the TTS operation_queue, audio_queue, audio_generation, or + // has_audio_source_atomic. + let handle = tokio::spawn(run_loading_loop(source, clip, token.clone())); + *loop_guard = Some(LoadingLoopHandle { handle, token }); + + info!("Started loading audio loop"); + Ok(()) + } + + /// Stops a running loading-audio loop, triggering its short fade-out. + /// + /// This is infallible, silent, and idempotent: with no loop running it does + /// nothing. The loop's handle is intentionally left in place so that + /// disconnect (or the next start) can await or replace it. + pub async fn stop_loading_audio(&self) { + let loop_guard = self.loading_loop.lock().await; + if let Some(existing) = loop_guard.as_ref() { + existing.token.cancel(); + debug!("Requested loading audio loop stop"); + } + } + + /// Test-only: clones the running loading loop's cancellation token, if any, + /// so a test can observe whether teardown (or `Drop`) cancelled it. + #[cfg(test)] + pub(crate) async fn loading_loop_token_for_test(&self) -> Option { + self.loading_loop + .lock() + .await + .as_ref() + .map(|existing| existing.token.clone()) + } +} + +/// Runs the loading-audio playback loop until cancelled, then fades out. +/// +/// The task holds only its `Arc`, `Arc`, and +/// `CancellationToken` — no locks are held during the loop, and no TTS state is +/// ever touched (the isolation invariant from the module docs). +async fn run_loading_loop( + source: Arc, + clip: Arc, + token: CancellationToken, +) { + let sample_rate = clip.sample_rate(); + let channels = clip.channels(); + let samples_per_channel = (sample_rate / 100) as usize; + let frame_len = loading_frame_len(sample_rate, channels); + + if frame_len == 0 || clip.samples().is_empty() { + warn!("Loading audio clip has no playable frames; loop not started"); + return; + } + + let mut frame_buf: Vec = vec![0; frame_len]; + let mut cursor: usize = 0; + + // Main loop: feed 10 ms frames until the cancellation token fires. Once the + // source's internal queue fills, `capture_frame().await` applies real-time + // back-pressure, pacing the loop at one 10 ms frame per 10 ms of wall-clock + // — so no manual timer is needed. + loop { + cursor = fill_loading_frame(clip.samples(), cursor, &mut frame_buf); + let frame = AudioFrame { + data: std::borrow::Cow::Borrowed(&frame_buf), + sample_rate, + num_channels: channels as u32, + samples_per_channel: samples_per_channel as u32, + }; + + tokio::select! { + biased; + _ = token.cancelled() => break, + res = source.capture_frame(&frame) => { + if res.is_err() { + debug!("Loading audio source closed; loop exiting without fade-out"); + return; + } + } + } + } + + // Fade-out phase: scale subsequent frames by a linear 1.0 -> 0.0 ramp so the + // loop stops without an audible click. The fade composes with the + // already-volume-scaled clip samples. + let fade_frames = (LOADING_AUDIO_FADE_OUT_MS / 10).max(1) as usize; + let total_fade_samples = fade_frames * frame_len; + let mut fade_index: usize = 0; + + for _ in 0..fade_frames { + cursor = fill_loading_frame(clip.samples(), cursor, &mut frame_buf); + fade_index = apply_fade(&mut frame_buf, fade_index, total_fade_samples); + let frame = AudioFrame { + data: std::borrow::Cow::Borrowed(&frame_buf), + sample_rate, + num_channels: channels as u32, + samples_per_channel: samples_per_channel as u32, + }; + if source.capture_frame(&frame).await.is_err() { + break; + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_loading_frame_len_mono_and_stereo() { + for rate in [8_000u32, 16_000, 24_000, 44_100, 48_000] { + let expected_mono = rate as usize / 100; + assert_eq!( + loading_frame_len(rate, 1), + expected_mono, + "mono frame length wrong at {rate} Hz" + ); + assert_eq!( + loading_frame_len(rate, 2), + expected_mono * 2, + "stereo frame length wrong at {rate} Hz" + ); + } + } + + #[test] + fn test_fill_loading_frame_cursor_wraps_to_expected_offset() { + // Clip of 10 samples; frames of 4 advance the cursor 4 each time. + let clip: Vec = (0..10).collect(); + let mut out = [0i16; 4]; + let mut cursor = 0usize; + for _ in 0..5 { + cursor = fill_loading_frame(&clip, cursor, &mut out); + } + // After 5 frames of 4 samples, 20 samples consumed; 20 % 10 == 0. + assert_eq!(cursor, 0); + } + + #[test] + fn test_fill_loading_frame_straddles_boundary_tail_then_head() { + // Clip [10, 20, 30, 40] (len 4); a frame of 3 starting at cursor 2 must + // yield the tail [30, 40] followed by the head [10]. The cursor advances + // 3 from 2, wrapping to (2 + 3) % 4 == 1. + let clip: Vec = vec![10, 20, 30, 40]; + let mut out = [0i16; 3]; + let cursor = fill_loading_frame(&clip, 2, &mut out); + assert_eq!(out, [30, 40, 10]); + assert_eq!(cursor, 1); + } + + #[test] + fn test_fill_loading_frame_reproduces_clip_contiguously() { + // Reading the clip in frames must reproduce its contents in order. + let clip: Vec = vec![1, 2, 3, 4, 5, 6]; + let mut reconstructed: Vec = Vec::new(); + let mut out = [0i16; 2]; + let mut cursor = 0usize; + for _ in 0..3 { + cursor = fill_loading_frame(&clip, cursor, &mut out); + reconstructed.extend_from_slice(&out); + } + assert_eq!(reconstructed, clip); + assert_eq!(cursor, 0); + + // Continuing wraps and reproduces the clip again from the start. + let mut second_pass: Vec = Vec::new(); + for _ in 0..3 { + cursor = fill_loading_frame(&clip, cursor, &mut out); + second_pass.extend_from_slice(&out); + } + assert_eq!(second_pass, clip); + } + + #[test] + fn test_fill_loading_frame_empty_clip_is_safe() { + let clip: Vec = Vec::new(); + let mut out = [7i16; 4]; + let cursor = fill_loading_frame(&clip, 3, &mut out); + assert_eq!(cursor, 0); + assert_eq!(out, [0, 0, 0, 0]); + } + + #[test] + fn test_fade_factor_monotonic_non_increasing_and_ends_at_zero() { + let total = 9usize; + let mut previous = f32::MAX; + for index in 0..total { + let value = fade_factor(index, total); + assert!((0.0..=1.0).contains(&value), "fade factor out of range"); + assert!(value <= previous, "fade factor not non-increasing"); + previous = value; + } + assert_eq!(fade_factor(total - 1, total), 0.0); + } + + #[test] + fn test_fade_factor_zero_total_is_safe() { + assert_eq!(fade_factor(0, 0), 0.0); + assert_eq!(fade_factor(5, 0), 0.0); + } + + #[test] + fn test_apply_fade_produces_decreasing_amplitude_ending_at_zero() { + // A constant-amplitude frame faded across the whole window must come out + // monotonically non-increasing in magnitude and end at exactly zero. + let total = 16usize; + let mut frame = vec![1000i16; total]; + let next = apply_fade(&mut frame, 0, total); + + assert_eq!(next, total, "fade index must advance by the frame length"); + assert_eq!(frame[total - 1], 0, "the last faded sample must be zero"); + assert!( + frame[0] > 0 && frame[0] <= 1000, + "the first faded sample is attenuated but still non-zero" + ); + for pair in frame.windows(2) { + assert!( + pair[1].abs() <= pair[0].abs(), + "fade amplitude must be monotonically non-increasing" + ); + } + } + + #[test] + fn test_apply_fade_continues_ramp_across_frames() { + // Splitting the fade across two frames must continue the ramp seamlessly: + // the second frame resumes where the first stopped and still ends at zero. + let total = 8usize; + let mut first = vec![2000i16; 4]; + let mut second = vec![2000i16; 4]; + + let next = apply_fade(&mut first, 0, total); + assert_eq!(next, 4); + let end = apply_fade(&mut second, next, total); + assert_eq!(end, total); + assert_eq!(second[3], 0, "the final faded sample must be zero"); + + let combined: Vec = first.iter().chain(second.iter()).copied().collect(); + for pair in combined.windows(2) { + assert!( + pair[1].abs() <= pair[0].abs(), + "fade amplitude must be non-increasing across the frame boundary" + ); + } + } + + #[tokio::test] + async fn test_run_loading_loop_exits_cleanly_when_precancelled() { + // A 500 ms 16 kHz mono clip, well above the minimum duration. + let samples: Vec = (0..8_000).map(|i| (i % 1000) as i16).collect(); + let data = crate::livekit::loading_clip::raw_pcm_to_base64(&samples); + let clip = Arc::new( + crate::livekit::decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect("raw PCM clip should decode"), + ); + + let source = Arc::new(NativeAudioSource::new( + AudioSourceOptions { + echo_cancellation: false, + noise_suppression: false, + auto_gain_control: false, + }, + clip.sample_rate(), + clip.channels() as u32, + LOADING_AUDIO_QUEUE_SIZE_MS, + )); + + // A token cancelled before the loop runs: the loop must skip the main + // phase, emit only the fade-out, and exit cleanly without an abort. + let token = CancellationToken::new(); + token.cancel(); + + let handle = tokio::spawn(run_loading_loop(source, clip, token)); + let outcome = tokio::time::timeout(std::time::Duration::from_secs(5), handle).await; + + assert!( + outcome.is_ok(), + "run_loading_loop must exit promptly when the token is pre-cancelled" + ); + assert!( + outcome.expect("loop completed within timeout").is_ok(), + "run_loading_loop task must not panic" + ); + } +} diff --git a/src/livekit/client/mod.rs b/src/livekit/client/mod.rs index cc8d663..b7163f2 100644 --- a/src/livekit/client/mod.rs +++ b/src/livekit/client/mod.rs @@ -9,6 +9,7 @@ mod audio; mod callbacks; mod connection; mod events; +mod loading_audio; mod messaging; mod operation_worker; @@ -27,6 +28,7 @@ use tracing::warn; pub(crate) use super::operations::{LiveKitOperation, OperationQueue, QueueStats}; pub(crate) use super::types::LiveKitConfig; +use crate::livekit::loading_clip::LoadingClip; /// Callback type for handling incoming audio chunks. pub type AudioCallback = Arc) + Send + Sync>; @@ -125,6 +127,22 @@ pub struct LiveKitClient { pub(crate) audio_source: Arc>>>, pub(crate) local_audio_track: Option, pub(crate) local_track_publication: Arc>>, + /// Decoded loading-indicator clip; `None` when the feature is unused. + pub(crate) loading_clip: Option>, + /// Dedicated audio source feeding the "loading-audio" track. + pub(crate) loading_audio_source: Arc>>>, + /// Handle keeping the "loading-audio" track published. + pub(crate) loading_audio_track: Option, + /// Publication of the dedicated "loading-audio" track. + pub(crate) loading_track_publication: Arc>>, + /// Running loading-audio loop handle and its cancellation token, if active. + /// + /// Wrapped in `Arc` so the operation worker's `OperationContext` can hold a + /// clone and cancel the loop from `process_reconnect` (the running loop's + /// audio source dies with the pre-reconnect room). Visibility is + /// `pub(super)` because [`loading_audio::LoadingLoopHandle`] is an internal + /// type only used within the `client` submodule. + pub(super) loading_loop: Arc>>, pub(crate) _audio_buffer_pool: Arc>>>, pub(crate) audio_queue: Arc>>>, pub(crate) operation_queue: Option, @@ -185,6 +203,11 @@ impl LiveKitClient { audio_source: Arc::new(Mutex::new(None)), local_audio_track: None, local_track_publication: Arc::new(Mutex::new(None)), + loading_clip: None, + loading_audio_source: Arc::new(Mutex::new(None)), + loading_audio_track: None, + loading_track_publication: Arc::new(Mutex::new(None)), + loading_loop: Arc::new(Mutex::new(None)), _audio_buffer_pool: Arc::new(Mutex::new( (0..4) .map(|_| Vec::with_capacity(buffer_capacity)) @@ -234,6 +257,31 @@ impl LiveKitClient { pub(crate) fn has_room_events(&self) -> bool { self.room_events.is_some() } + + #[cfg(test)] + pub(crate) fn has_loading_clip(&self) -> bool { + self.loading_clip.is_some() + } + + #[cfg(test)] + pub(crate) async fn has_loading_audio_source(&self) -> bool { + self.loading_audio_source.lock().await.is_some() + } + + #[cfg(test)] + pub(crate) fn has_loading_audio_track(&self) -> bool { + self.loading_audio_track.is_some() + } + + #[cfg(test)] + pub(crate) async fn has_loading_track_publication(&self) -> bool { + self.loading_track_publication.lock().await.is_some() + } + + #[cfg(test)] + pub(crate) async fn has_loading_loop(&self) -> bool { + self.loading_loop.lock().await.is_some() + } } impl Drop for LiveKitClient { @@ -241,5 +289,19 @@ impl Drop for LiveKitClient { if self.operation_worker_handle.is_some() { warn!("LiveKitClient dropped without explicit disconnect call"); } + + // Backstop for the loading-audio loop. `disconnect` is the normal + // teardown path, but session cleanup may skip it if the LiveKit write + // lock cannot be acquired in time. A buffered `NativeAudioSource` keeps + // accepting frames even after the room is gone, so the loop would never + // observe a `capture_frame` error and never self-terminate. Cancelling + // the token and aborting the task here keeps the loop from being + // orphaned regardless of how the client is dropped. + if let Ok(mut guard) = self.loading_loop.try_lock() + && let Some(loading) = guard.take() + { + loading.token.cancel(); + loading.handle.abort(); + } } } diff --git a/src/livekit/client/operation_worker.rs b/src/livekit/client/operation_worker.rs index 194f312..7831802 100644 --- a/src/livekit/client/operation_worker.rs +++ b/src/livekit/client/operation_worker.rs @@ -12,11 +12,13 @@ use livekit::webrtc::audio_source::native::NativeAudioSource; use tokio::sync::Mutex; use tracing::{debug, info, warn}; +use super::loading_audio::LoadingLoopHandle; use super::{ AudioCallback, DataCallback, LiveKitClient, LiveKitConfig, LiveKitOperation, OperationQueue, ParticipantConnectCallback, ParticipantDisconnectCallback, TrackSubscribedCallback, }; use crate::AppError; +use crate::livekit::loading_clip::LoadingClip; /// Context struct for process_operation to avoid too many arguments pub(super) struct OperationContext { @@ -35,6 +37,20 @@ pub(super) struct OperationContext { pub(super) track_subscribed_callback: Option, /// Generation counter - audio operations with lower generation are skipped pub(super) audio_generation: Arc, + // The following loading-audio fields are read by `process_reconnect` to + // rebuild the dedicated "loading-audio" track after a reconnect, and to + // cancel a loop left running on the now-dead pre-reconnect source. They do + // NOT entangle the loading loop with TTS queue/generation state — the + // loading loop's isolation invariant still holds. + /// Decoded loading-indicator clip; `None` when the feature is unused. + pub(super) loading_clip: Option>, + /// Dedicated audio source feeding the "loading-audio" track. + pub(super) loading_audio_source: Arc>>>, + /// Publication of the dedicated "loading-audio" track. + pub(super) loading_track_publication: Arc>>, + /// Running loading-audio loop, so `process_reconnect` can cancel a loop + /// whose audio source died with the pre-reconnect room. + pub(super) loading_loop: Arc>>, } impl LiveKitClient { @@ -57,6 +73,10 @@ impl LiveKitClient { participant_connect_callback: self.participant_connect_callback.clone(), track_subscribed_callback: self.track_subscribed_callback.clone(), audio_generation, + loading_clip: self.loading_clip.clone(), + loading_audio_source: Arc::clone(&self.loading_audio_source), + loading_track_publication: Arc::clone(&self.loading_track_publication), + loading_loop: Arc::clone(&self.loading_loop), }; let stats = Arc::clone(&self.stats); diff --git a/src/livekit/client/tests.rs b/src/livekit/client/tests.rs index 53fc4dc..8c6e90d 100644 --- a/src/livekit/client/tests.rs +++ b/src/livekit/client/tests.rs @@ -318,3 +318,269 @@ async fn test_livekit_client_operation_priority_ordering() { let queue = client.get_operation_queue(); assert!(queue.is_none()); // Queue only exists after connect } + +/// Builds a base64-encoded raw 16-bit PCM clip long enough to pass the +/// minimum-duration validation (500 ms of 16 kHz mono audio). +fn make_loading_clip_base64() -> String { + let samples: Vec = (0..8_000).map(|i| (i % 1000) as i16).collect(); + crate::livekit::loading_clip::raw_pcm_to_base64(&samples) +} + +#[tokio::test] +async fn test_livekit_client_loading_fields_default() { + let config = create_test_config(); + let client = LiveKitClient::new(config); + + assert!(!client.has_loading_clip()); + assert!(!client.has_loading_audio_source().await); + assert!(!client.has_loading_audio_track()); + assert!(!client.has_loading_track_publication().await); + assert!(!client.has_loading_loop().await); +} + +#[tokio::test] +async fn test_set_loading_audio_clip_stores_clip() { + let data = make_loading_clip_base64(); + let clip = crate::livekit::decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect("raw PCM loading clip should decode"); + + let config = create_test_config(); + let mut client = LiveKitClient::new(config); + assert!(!client.has_loading_clip()); + + client.set_loading_audio_clip(clip); + assert!(client.has_loading_clip()); +} + +#[tokio::test] +async fn test_setup_loading_audio_track_without_clip_is_noop() { + let mut client = LiveKitClient::new(create_test_config()); + + // With no loading clip configured, track setup is a clean no-op — it + // returns Ok and publishes nothing, leaving every loading field empty. + client + .setup_loading_audio_track() + .await + .expect("setup_loading_audio_track must be a no-op when no clip is configured"); + + assert!(!client.has_loading_audio_source().await); + assert!(!client.has_loading_audio_track()); + assert!(!client.has_loading_track_publication().await); +} + +#[tokio::test] +async fn test_setup_loading_audio_track_with_clip_requires_room() { + let data = make_loading_clip_base64(); + let clip = crate::livekit::decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect("raw PCM loading clip should decode"); + + let mut client = LiveKitClient::new(create_test_config()); + client.set_loading_audio_clip(clip); + + // A clip is configured but no room is connected: setup must surface an + // error rather than silently succeed, confirming the clip-present path is + // taken and the track is not published without a room. + let result = client.setup_loading_audio_track().await; + assert!( + result.is_err(), + "setup_loading_audio_track must error when a clip is set but no room is available" + ); + assert!(!client.has_loading_audio_source().await); + assert!(!client.has_loading_track_publication().await); +} + +#[tokio::test] +async fn test_stop_loading_audio_no_loop_is_noop() { + let config = create_test_config(); + let client = LiveKitClient::new(config); + + // Stopping with no loop running must not panic and must leave no loop. + client.stop_loading_audio().await; + assert!(!client.has_loading_loop().await); +} + +#[tokio::test] +async fn test_start_loading_audio_not_connected_errors() { + let config = create_test_config(); + let client = LiveKitClient::new(config); + + let result = client.start_loading_audio().await; + assert!( + result.is_err(), + "start_loading_audio should fail when not connected" + ); +} + +#[tokio::test] +async fn test_start_loading_audio_connected_without_clip_errors() { + let config = create_test_config(); + let client = LiveKitClient::new(config); + client.set_connected(true).await; + + let result = client.start_loading_audio().await; + assert!( + result.is_err(), + "start_loading_audio should fail without a loading source/clip" + ); +} + +#[tokio::test] +async fn test_disconnect_clears_loading_fields() { + let config = create_test_config(); + let mut client = LiveKitClient::new(config); + client.set_connected(true).await; + + assert!(client.disconnect().await.is_ok()); + + assert!(!client.has_loading_clip()); + assert!(!client.has_loading_audio_source().await); + assert!(!client.has_loading_audio_track()); + assert!(!client.has_loading_track_publication().await); + assert!(!client.has_loading_loop().await); +} + +#[tokio::test] +async fn test_two_independent_native_audio_sources() { + use livekit::track::LocalAudioTrack; + use livekit::webrtc::audio_source::native::NativeAudioSource; + use livekit::webrtc::prelude::{AudioSourceOptions, RtcAudioSource}; + + // `AudioSourceOptions` is not `Clone`; construct one per source. + let tts_options = AudioSourceOptions { + echo_cancellation: false, + noise_suppression: false, + auto_gain_control: false, + }; + let loading_options = AudioSourceOptions { + echo_cancellation: false, + noise_suppression: false, + auto_gain_control: false, + }; + + // TTS-style source at 24 kHz mono. + let tts_source = NativeAudioSource::new(tts_options, 24_000, 1, 240); + let tts_track = + LocalAudioTrack::create_audio_track("tts-audio", RtcAudioSource::Native(tts_source)); + + // Loading-style source at a different sample rate, 16 kHz mono. + let loading_source = NativeAudioSource::new(loading_options, 16_000, 1, 160); + let loading_track = LocalAudioTrack::create_audio_track( + "loading-audio", + RtcAudioSource::Native(loading_source), + ); + + // Both tracks must construct without panic, confirming a participant can + // hold two independent audio sources/tracks at different sample rates. + assert_eq!(tts_track.name(), "tts-audio"); + assert_eq!(loading_track.name(), "loading-audio"); +} + +/// Builds a connected `LiveKitClient` with a loading clip set and a loading +/// audio source injected, ready to exercise the loading loop. The source is +/// injected directly because `setup_loading_audio_track` needs a live room, +/// which unit tests do not have. +async fn connected_client_with_loading_source() -> LiveKitClient { + use livekit::webrtc::audio_source::native::NativeAudioSource; + use livekit::webrtc::prelude::AudioSourceOptions; + + let data = make_loading_clip_base64(); + let clip = crate::livekit::decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect("raw PCM loading clip should decode"); + + let mut client = LiveKitClient::new(create_test_config()); + client.set_connected(true).await; + client.set_loading_audio_clip(clip); + + let source = Arc::new(NativeAudioSource::new( + AudioSourceOptions { + echo_cancellation: false, + noise_suppression: false, + auto_gain_control: false, + }, + 16_000, + 1, + super::loading_audio::LOADING_AUDIO_QUEUE_SIZE_MS, + )); + *client.loading_audio_source.lock().await = Some(source); + client +} + +#[tokio::test] +async fn test_start_loading_audio_idempotent_and_disconnect_tears_down() { + let mut client = connected_client_with_loading_source().await; + + // The first start spawns the loop. + client + .start_loading_audio() + .await + .expect("first start_loading_audio should succeed"); + assert!(client.has_loading_loop().await); + + // A second start while the loop is still running is an idempotent no-op. + client + .start_loading_audio() + .await + .expect("second start_loading_audio should be an idempotent no-op"); + assert!(client.has_loading_loop().await); + + // Disconnect cancels the loop, awaits it (with the abort backstop), and + // clears every loading field. + client + .disconnect() + .await + .expect("disconnect should succeed"); + assert!(!client.has_loading_loop().await); + assert!(!client.has_loading_audio_source().await); + assert!(!client.has_loading_clip()); +} + +#[tokio::test] +async fn test_rapid_start_stop_loading_audio_is_clean() { + let mut client = connected_client_with_loading_source().await; + + // Rapidly alternating start/stop must never panic, and a disconnect + // afterwards (with the loop possibly mid-fade) must still tear down cleanly + // without hanging. + for _ in 0..30 { + client + .start_loading_audio() + .await + .expect("start_loading_audio should not error"); + client.stop_loading_audio().await; + } + + let teardown = timeout(Duration::from_secs(5), client.disconnect()).await; + assert!( + teardown.is_ok(), + "disconnect must not hang after rapid start/stop" + ); + assert!(teardown.expect("disconnect completed").is_ok()); + assert!(!client.has_loading_loop().await); + assert!(!client.has_loading_audio_source().await); +} + +#[tokio::test] +async fn test_drop_cancels_active_loading_loop() { + let client = connected_client_with_loading_source().await; + client + .start_loading_audio() + .await + .expect("start_loading_audio should succeed"); + + // Hold a clone of the running loop's cancellation token, then drop the + // client WITHOUT calling disconnect — the path session cleanup takes when + // it cannot acquire the LiveKit lock in time. + let token = client + .loading_loop_token_for_test() + .await + .expect("a loop should be running"); + assert!(!token.is_cancelled()); + + drop(client); + + // The Drop backstop must have cancelled the loop so it cannot be orphaned. + assert!( + token.is_cancelled(), + "dropping the client must cancel the loading loop" + ); +} diff --git a/src/livekit/loading_clip.rs b/src/livekit/loading_clip.rs new file mode 100644 index 0000000..fa6f697 --- /dev/null +++ b/src/livekit/loading_clip.rs @@ -0,0 +1,704 @@ +//! # Loading Clip Decode Layer +//! +//! Pure decoding and validation for the "loading indicator" audio clip — a short +//! audio sample played back to a caller while the system is busy (for example, +//! while a slow tool or model call is in flight). +//! +//! This module is intentionally a self-contained decode/validation layer. It +//! depends only on `std`, [`hound`], and [`base64`]; it knows nothing about +//! WebSocket messages or LiveKit-client types. Callers pass a base64-encoded +//! audio payload (WAV or raw PCM) and receive a fully decoded, validated, and +//! volume-scaled [`LoadingClip`] ready for playback. + +use base64::{Engine as _, engine::general_purpose}; + +/// Minimum accepted duration of a loading clip, in milliseconds. +/// +/// Clips shorter than this are rejected because they are too brief to be a +/// meaningful audible indicator. +pub const MIN_LOADING_AUDIO_MS: u32 = 250; + +/// Maximum accepted duration of a loading clip, in milliseconds. +/// +/// Clips longer than this are rejected to bound memory use and avoid playing an +/// excessively long sample. +pub const MAX_LOADING_AUDIO_MS: u32 = 10_000; + +/// Lowest accepted clip sample rate, in Hz. +pub const LOADING_AUDIO_SAMPLE_RATE_MIN: u32 = 8_000; + +/// Highest accepted clip sample rate, in Hz. +pub const LOADING_AUDIO_SAMPLE_RATE_MAX: u32 = 48_000; + +/// Maximum accepted size of the decoded audio payload, in bytes. +/// +/// Derived as the worst-case size of a clip at the maximum sample rate, with +/// stereo channels, 16-bit (2 bytes per sample) PCM, lasting the maximum +/// allowed duration. This evaluates to `1_920_000` bytes and is used as an +/// early guard to reject oversized payloads before fully decoding them. +pub const MAX_LOADING_AUDIO_DECODED_BYTES: usize = (LOADING_AUDIO_SAMPLE_RATE_MAX as usize) + * 2 /* max channels */ + * 2 /* bytes per i16 */ + * (MAX_LOADING_AUDIO_MS as usize) + / 1000; + +/// A decoded, validated, ready-to-play loading audio sample. +/// +/// The PCM data in [`LoadingClip::samples`] is flat and interleaved when +/// stereo, and already has the requested volume scaling applied. Construct one +/// via [`decode_loading_clip`]. +#[derive(Clone)] +pub struct LoadingClip { + /// Decoded PCM, flat and interleaved when stereo, with `volume` applied. + samples: Vec, + /// The clip's native sample rate, in Hz. + sample_rate: u32, + /// Number of channels: `1` (mono) or `2` (stereo). + channels: u16, + /// The volume (in `0.0..=1.0`) that was applied to `samples`. + volume: f32, +} + +impl std::fmt::Debug for LoadingClip { + /// Formats the clip without dumping the (potentially ~1 MB) sample buffer; + /// only the sample count and metadata are shown. + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("LoadingClip") + .field("sample_count", &self.samples.len()) + .field("sample_rate", &self.sample_rate) + .field("channels", &self.channels) + .field("volume", &self.volume) + .finish() + } +} + +impl LoadingClip { + /// Returns the decoded PCM samples, flat and interleaved when stereo, with + /// volume scaling already applied. + pub fn samples(&self) -> &[i16] { + &self.samples + } + + /// Returns the clip's native sample rate, in Hz. + pub fn sample_rate(&self) -> u32 { + self.sample_rate + } + + /// Returns the number of channels: `1` (mono) or `2` (stereo). + pub fn channels(&self) -> u16 { + self.channels + } + + /// Returns the volume (in `0.0..=1.0`) that was applied to the samples. + pub fn volume(&self) -> f32 { + self.volume + } +} + +/// Applies linear volume scaling to PCM samples without overflow. +/// +/// Each sample is widened to `f32`, multiplied by `volume`, rounded, and +/// clamped back into the `i16` range so that no value can wrap or overflow. +fn apply_volume(samples: &mut [i16], volume: f32) { + for sample in samples.iter_mut() { + let scaled = (f32::from(*sample) * volume).round(); + *sample = scaled.clamp(f32::from(i16::MIN), f32::from(i16::MAX)) as i16; + } +} + +/// Decodes and validates a base64-encoded loading audio clip. +/// +/// `data` is the base64-encoded payload. `format` is an optional explicit +/// container hint (`"wav"` or `"pcm"`); when `None` the format is auto-detected +/// from the decoded bytes. `sample_rate` and `channels` describe raw PCM input +/// and are ignored for WAV input (the WAV header is authoritative). `volume` is +/// an optional playback gain in `0.0..=1.0`; out-of-range values are clamped +/// rather than rejected, and the applied value is recorded on the returned clip. +/// +/// On success returns a fully decoded, validated, volume-scaled [`LoadingClip`]. +/// On failure returns an `Err` with a human-readable, end-user-facing message. +pub fn decode_loading_clip( + data: &str, + format: Option<&str>, + sample_rate: Option, + channels: Option, + volume: Option, +) -> Result { + // 1. Early size guard based on the base64 length (~3 decoded bytes per 4 + // encoded characters), so an obviously oversized payload is rejected + // before allocating the decoded buffer. + let estimated_decoded_bytes = data.len() / 4 * 3; + if estimated_decoded_bytes > MAX_LOADING_AUDIO_DECODED_BYTES { + return Err(format!( + "loading_audio.data decoded audio would exceed the maximum size of \ + {MAX_LOADING_AUDIO_DECODED_BYTES} bytes" + )); + } + + // 2. Base64 decode. + let bytes = general_purpose::STANDARD + .decode(data) + .map_err(|_| "loading_audio.data is not valid base64".to_string())?; + + // 3. Post-decode size guard against the exact decoded length. + if bytes.len() > MAX_LOADING_AUDIO_DECODED_BYTES { + return Err(format!( + "loading_audio.data decoded audio exceeds the maximum size of \ + {MAX_LOADING_AUDIO_DECODED_BYTES} bytes" + )); + } + + // 4. Format detection: explicit hint wins, otherwise auto-detect. + let is_wav = match format { + Some(fmt) => match fmt.trim().to_lowercase().as_str() { + "wav" => true, + "pcm" => false, + _ => { + return Err("loading_audio.format must be \"wav\" or \"pcm\"".to_string()); + } + }, + None => bytes.len() >= 12 && &bytes[0..4] == b"RIFF" && &bytes[8..12] == b"WAVE", + }; + + // 5/6. Decode into PCM samples plus the authoritative sample rate and + // channel count for the chosen container. + let (mut samples, sample_rate, channels) = if is_wav { + decode_wav(&bytes)? + } else { + decode_raw_pcm(&bytes, sample_rate, channels)? + }; + + // 7. Validate channel count. + if channels != 1 && channels != 2 { + return Err("loading_audio supports mono or stereo only".to_string()); + } + + // 8. Validate sample rate. + if !(LOADING_AUDIO_SAMPLE_RATE_MIN..=LOADING_AUDIO_SAMPLE_RATE_MAX).contains(&sample_rate) { + return Err(format!( + "loading_audio sample rate must be between {LOADING_AUDIO_SAMPLE_RATE_MIN} Hz \ + and {LOADING_AUDIO_SAMPLE_RATE_MAX} Hz" + )); + } + + // 9. Validate duration. + let frames = samples.len() / channels as usize; + if samples.is_empty() || frames == 0 { + return Err(format!( + "loading_audio is too short; the minimum duration is {MIN_LOADING_AUDIO_MS} ms" + )); + } + let duration_ms = frames as u64 * 1000 / sample_rate as u64; + if duration_ms < MIN_LOADING_AUDIO_MS as u64 { + return Err(format!( + "loading_audio is too short; the minimum duration is {MIN_LOADING_AUDIO_MS} ms" + )); + } + if duration_ms > MAX_LOADING_AUDIO_MS as u64 { + return Err(format!( + "loading_audio is too long; the maximum duration is {MAX_LOADING_AUDIO_MS} ms" + )); + } + + // 10. Apply volume scaling. Out-of-range values are clamped, not rejected. + let volume = volume.unwrap_or(1.0).clamp(0.0, 1.0); + if (volume - 1.0).abs() > f32::EPSILON { + apply_volume(&mut samples, volume); + } + + // 11. Done. + Ok(LoadingClip { + samples, + sample_rate, + channels, + volume, + }) +} + +/// Decodes a 16-bit PCM WAV payload, returning its samples, sample rate, and +/// channel count taken from the (authoritative) WAV header. +fn decode_wav(bytes: &[u8]) -> Result<(Vec, u32, u16), String> { + let mut reader = hound::WavReader::new(std::io::Cursor::new(bytes)) + .map_err(|e| format!("loading_audio is not a valid WAV file: {e}"))?; + + let spec = reader.spec(); + if spec.sample_format != hound::SampleFormat::Int || spec.bits_per_sample != 16 { + return Err( + "loading_audio: only 16-bit PCM is supported; please provide 16-bit PCM audio" + .to_string(), + ); + } + + // Guard against a forged WAV `data`-chunk length. `hound`'s sample + // iterator pre-allocates its `Vec` from the header's declared sample + // count, so a tiny file claiming an enormous `data` chunk would force a + // multi-gigabyte allocation before a single sample byte is read. The + // base64-length and decoded-byte guards in `decode_loading_clip` cannot + // catch this: the payload is small on the wire but its header claims to + // decode huge. Reject using the declared count, before `collect()`. + let declared_bytes = (reader.len() as usize).saturating_mul(2); + if declared_bytes > MAX_LOADING_AUDIO_DECODED_BYTES { + return Err(format!( + "loading_audio.data decoded audio exceeds the maximum size of \ + {MAX_LOADING_AUDIO_DECODED_BYTES} bytes" + )); + } + + let samples = reader + .samples::() + .collect::, _>>() + .map_err(|e| format!("loading_audio could not be read as 16-bit PCM: {e}"))?; + + // Reject a malformed WAV whose sample count is not a whole number of + // interleaved frames, mirroring the raw-PCM frame-alignment check. + if spec.channels == 0 || !samples.len().is_multiple_of(spec.channels as usize) { + return Err("loading_audio WAV data is not a whole number of audio frames".to_string()); + } + + Ok((samples, spec.sample_rate, spec.channels)) +} + +/// Decodes a little-endian raw 16-bit PCM payload using the caller-supplied +/// `sample_rate` and `channels`, validating that the byte length is +/// frame-aligned. +fn decode_raw_pcm( + bytes: &[u8], + sample_rate: Option, + channels: Option, +) -> Result<(Vec, u32, u16), String> { + let sample_rate = sample_rate + .ok_or_else(|| "sample_rate is required for raw PCM loading_audio".to_string())?; + let channels = channels.unwrap_or(1); + + // `channels` defaults to 1 and is validated against {1, 2} by the caller, + // so `frame_bytes` is always non-zero (>= 2) here. + let frame_bytes = 2 * channels as usize; + if !bytes.len().is_multiple_of(2) || !bytes.len().is_multiple_of(frame_bytes) { + return Err("loading_audio raw PCM length is not frame-aligned".to_string()); + } + + let samples = bytes + .chunks_exact(2) + .map(|c| i16::from_le_bytes([c[0], c[1]])) + .collect(); + + Ok((samples, sample_rate, channels)) +} + +/// Encodes interleaved 16-bit samples as little-endian raw PCM, then base64. +/// +/// Test-only helper shared by the loading-audio test suites in this crate. +#[cfg(test)] +pub(crate) fn raw_pcm_to_base64(samples: &[i16]) -> String { + let mut bytes = Vec::with_capacity(samples.len() * 2); + for sample in samples { + bytes.extend_from_slice(&sample.to_le_bytes()); + } + general_purpose::STANDARD.encode(bytes) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Builds an in-memory 16-bit-PCM WAV with the given sample rate, channel + /// count, and interleaved samples, returning its base64 encoding. + fn make_wav_base64(sample_rate: u32, channels: u16, samples: &[i16]) -> String { + let spec = hound::WavSpec { + channels, + sample_rate, + bits_per_sample: 16, + sample_format: hound::SampleFormat::Int, + }; + let mut cursor = std::io::Cursor::new(Vec::::new()); + { + let mut writer = + hound::WavWriter::new(&mut cursor, spec).expect("WAV writer should be created"); + for sample in samples { + writer + .write_sample(*sample) + .expect("sample should be written"); + } + writer.finalize().expect("WAV writer should finalize"); + } + general_purpose::STANDARD.encode(cursor.into_inner()) + } + + /// Builds an in-memory WAV with an arbitrary bit depth and sample format so + /// that 24-bit and 32-bit-float rejection paths can be exercised. Samples + /// are written as `i32` values; for the float format they are written via + /// `write_sample` with an `f32` cast. + fn make_wav_base64_with_format( + sample_rate: u32, + channels: u16, + bits_per_sample: u16, + sample_format: hound::SampleFormat, + frame_count: usize, + ) -> String { + let spec = hound::WavSpec { + channels, + sample_rate, + bits_per_sample, + sample_format, + }; + let mut cursor = std::io::Cursor::new(Vec::::new()); + { + let mut writer = + hound::WavWriter::new(&mut cursor, spec).expect("WAV writer should be created"); + for i in 0..(frame_count * channels as usize) { + match sample_format { + hound::SampleFormat::Float => { + let value = ((i % 100) as f32 / 100.0) - 0.5; + writer + .write_sample(value) + .expect("float sample should be written"); + } + hound::SampleFormat::Int => { + writer + .write_sample((i % 1000) as i32) + .expect("int sample should be written"); + } + } + } + writer.finalize().expect("WAV writer should finalize"); + } + general_purpose::STANDARD.encode(cursor.into_inner()) + } + + #[test] + fn test_valid_mono_wav_decodes() { + // 8000 samples at 16 kHz mono == 500 ms. + let samples: Vec = (0..8000).map(|i| (i % 1000) as i16).collect(); + let data = make_wav_base64(16_000, 1, &samples); + + let clip = decode_loading_clip(&data, None, None, None, None) + .expect("valid mono WAV should decode"); + + assert_eq!(clip.samples().len(), 8000); + assert_eq!(clip.sample_rate(), 16_000); + assert_eq!(clip.channels(), 1); + } + + #[test] + fn test_valid_stereo_wav_decodes() { + // 8000 frames * 2 channels == 16000 interleaved samples, 500 ms at 16 kHz. + let samples: Vec = (0..16_000).map(|i| (i % 1000) as i16).collect(); + let data = make_wav_base64(16_000, 2, &samples); + + let clip = decode_loading_clip(&data, None, None, None, None) + .expect("valid stereo WAV should decode"); + + assert_eq!(clip.channels(), 2); + assert_eq!(clip.samples().len(), 16_000); + assert_eq!(clip.sample_rate(), 16_000); + } + + #[test] + fn test_valid_raw_pcm_decodes() { + let samples: Vec = (0..8000).map(|i| (i % 500) as i16).collect(); + let data = raw_pcm_to_base64(&samples); + + let clip = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect("valid raw PCM should decode"); + + assert_eq!(clip.samples().len(), 8000); + assert_eq!(clip.sample_rate(), 16_000); + assert_eq!(clip.channels(), 1); + } + + #[test] + fn test_raw_pcm_without_sample_rate_rejected() { + let samples: Vec = (0..8000).map(|i| (i % 500) as i16).collect(); + let data = raw_pcm_to_base64(&samples); + + let err = decode_loading_clip(&data, Some("pcm"), None, Some(1), None) + .expect_err("raw PCM without sample_rate should be rejected"); + + assert_eq!(err, "sample_rate is required for raw PCM loading_audio"); + } + + #[test] + fn test_auto_detection_wav_and_raw_pcm() { + // WAV bytes without a format hint are detected as WAV. + let wav_samples: Vec = (0..8000).map(|i| (i % 1000) as i16).collect(); + let wav_data = make_wav_base64(16_000, 1, &wav_samples); + let wav_clip = decode_loading_clip(&wav_data, None, None, None, None) + .expect("WAV bytes should auto-detect as WAV"); + assert_eq!(wav_clip.sample_rate(), 16_000); + + // Non-WAV bytes without a format hint are treated as raw PCM. + let pcm_samples: Vec = (0..8000).map(|i| (i % 500) as i16).collect(); + let pcm_data = raw_pcm_to_base64(&pcm_samples); + let pcm_clip = decode_loading_clip(&pcm_data, None, Some(16_000), Some(1), None) + .expect("non-WAV bytes should auto-detect as raw PCM"); + assert_eq!(pcm_clip.samples().len(), 8000); + } + + #[test] + fn test_explicit_format_overrides_detection() { + // Raw PCM samples whose bytes do not form a RIFF/WAVE header, but the + // caller forces the "pcm" format anyway. + let pcm_samples: Vec = (0..8000).map(|i| (i % 500) as i16).collect(); + let pcm_data = raw_pcm_to_base64(&pcm_samples); + let clip = decode_loading_clip(&pcm_data, Some("pcm"), Some(16_000), Some(1), None) + .expect("explicit pcm format should decode raw PCM"); + assert_eq!(clip.samples().len(), 8000); + + // Forcing "wav" on raw PCM bytes fails WAV parsing. + let err = decode_loading_clip(&pcm_data, Some("wav"), Some(16_000), Some(1), None) + .expect_err("explicit wav format on raw PCM should fail"); + assert!( + err.contains("not a valid WAV file"), + "unexpected error: {err}" + ); + } + + #[test] + fn test_invalid_base64_rejected() { + let err = decode_loading_clip("not valid base64 @@@", None, None, None, None) + .expect_err("invalid base64 should be rejected"); + assert_eq!(err, "loading_audio.data is not valid base64"); + } + + #[test] + fn test_24bit_wav_rejected() { + // 8000 frames at 16 kHz keeps the duration valid so the 16-bit check + // is what triggers the rejection. + let data = make_wav_base64_with_format(16_000, 1, 24, hound::SampleFormat::Int, 8000); + let err = decode_loading_clip(&data, None, None, None, None) + .expect_err("24-bit WAV should be rejected"); + assert!(err.contains("16-bit"), "unexpected error: {err}"); + } + + #[test] + fn test_32bit_float_wav_rejected() { + let data = make_wav_base64_with_format(16_000, 1, 32, hound::SampleFormat::Float, 8000); + let err = decode_loading_clip(&data, None, None, None, None) + .expect_err("32-bit float WAV should be rejected"); + assert!(err.contains("16-bit"), "unexpected error: {err}"); + } + + #[test] + fn test_oversized_input_rejected() { + // Raw PCM larger than MAX_LOADING_AUDIO_DECODED_BYTES. + let oversized_samples = MAX_LOADING_AUDIO_DECODED_BYTES; // each i16 == 2 bytes + let samples: Vec = vec![0; oversized_samples]; + let data = raw_pcm_to_base64(&samples); + + let err = decode_loading_clip(&data, Some("pcm"), Some(48_000), Some(1), None) + .expect_err("oversized input should be rejected"); + assert!(err.contains("maximum size"), "unexpected error: {err}"); + } + + #[test] + fn test_too_short_clip_rejected() { + // 1000 samples at 16 kHz mono == ~62 ms, below MIN_LOADING_AUDIO_MS. + let samples: Vec = vec![100; 1000]; + let data = raw_pcm_to_base64(&samples); + + let err = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect_err("too-short clip should be rejected"); + assert!(err.contains("too short"), "unexpected error: {err}"); + } + + #[test] + fn test_too_long_clip_rejected() { + // 11 seconds at 16 kHz mono exceeds MAX_LOADING_AUDIO_MS (10 s) while + // staying within the decoded byte limit (352000 bytes < 1_920_000). + let samples: Vec = vec![100; 16_000 * 11]; + let data = raw_pcm_to_base64(&samples); + + let err = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect_err("too-long clip should be rejected"); + assert!(err.contains("too long"), "unexpected error: {err}"); + } + + #[test] + fn test_odd_length_raw_pcm_rejected() { + // An odd byte count cannot form whole 16-bit samples. + let bytes = vec![1u8, 2, 3]; + let data = general_purpose::STANDARD.encode(bytes); + + let err = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect_err("odd-length raw PCM should be rejected"); + assert_eq!(err, "loading_audio raw PCM length is not frame-aligned"); + } + + #[test] + fn test_non_frame_aligned_stereo_raw_pcm_rejected() { + // 8001 samples is even in bytes but not divisible by 2 channels. + let samples: Vec = vec![100; 8001]; + let data = raw_pcm_to_base64(&samples); + + let err = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(2), None) + .expect_err("non-frame-aligned stereo raw PCM should be rejected"); + assert_eq!(err, "loading_audio raw PCM length is not frame-aligned"); + } + + #[test] + fn test_sample_rate_too_low_rejected() { + // 4 kHz is below LOADING_AUDIO_SAMPLE_RATE_MIN. + let samples: Vec = vec![100; 8000]; + let data = raw_pcm_to_base64(&samples); + + let err = decode_loading_clip(&data, Some("pcm"), Some(4_000), Some(1), None) + .expect_err("too-low sample rate should be rejected"); + assert!(err.contains("sample rate"), "unexpected error: {err}"); + } + + #[test] + fn test_sample_rate_too_high_rejected() { + // 96 kHz is above LOADING_AUDIO_SAMPLE_RATE_MAX. 8000 samples keeps the + // duration valid so the sample-rate check is what triggers rejection. + let samples: Vec = vec![100; 8000]; + let data = raw_pcm_to_base64(&samples); + + let err = decode_loading_clip(&data, Some("pcm"), Some(96_000), Some(1), None) + .expect_err("too-high sample rate should be rejected"); + assert!(err.contains("sample rate"), "unexpected error: {err}"); + } + + #[test] + fn test_volume_half_scales_samples() { + let samples: Vec = vec![1000; 8000]; + let data = raw_pcm_to_base64(&samples); + + let clip = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), Some(0.5)) + .expect("clip with volume 0.5 should decode"); + + for sample in clip.samples() { + // 1000 * 0.5 == 500, within rounding tolerance. + assert!((*sample - 500).abs() <= 1, "sample not halved: {sample}"); + } + assert!((clip.volume() - 0.5).abs() < f32::EPSILON); + } + + #[test] + fn test_volume_one_and_none_leave_samples_unchanged() { + let samples: Vec = (0..8000).map(|i| (i % 1000) as i16).collect(); + let data = raw_pcm_to_base64(&samples); + + let clip_one = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), Some(1.0)) + .expect("clip with volume 1.0 should decode"); + assert_eq!(clip_one.samples(), samples.as_slice()); + + let clip_none = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect("clip with no volume should decode"); + assert_eq!(clip_none.samples(), samples.as_slice()); + } + + #[test] + fn test_out_of_range_volume_is_clamped() { + let samples: Vec = (0..8000).map(|i| (i % 1000) as i16).collect(); + let data = raw_pcm_to_base64(&samples); + + // Above 1.0 is clamped to 1.0 (samples unchanged), not an error. + let clip_high = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), Some(2.0)) + .expect("volume 2.0 should be clamped, not rejected"); + assert!((clip_high.volume() - 1.0).abs() < f32::EPSILON); + assert_eq!(clip_high.samples(), samples.as_slice()); + + // Below 0.0 is clamped to 0.0 (samples silenced), not an error. + let clip_low = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), Some(-1.0)) + .expect("volume -1.0 should be clamped, not rejected"); + assert!(clip_low.volume().abs() < f32::EPSILON); + assert!(clip_low.samples().iter().all(|s| *s == 0)); + } + + #[test] + fn test_volume_scaling_never_overflows() { + // Extreme input samples with volume near 1.0 must not wrap/overflow. + let mut samples: Vec = Vec::with_capacity(8000); + for i in 0..8000 { + samples.push(if i % 2 == 0 { i16::MAX } else { i16::MIN }); + } + let data = raw_pcm_to_base64(&samples); + + let clip = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), Some(0.999_999)) + .expect("extreme samples should decode"); + + for (i, sample) in clip.samples().iter().enumerate() { + // Compare magnitudes via i32 so i16::MIN's absolute value is + // representable. Attenuation below 1.0 must never flip the sign or + // grow the magnitude beyond the original (which would mean a wrap). + let scaled_abs = i32::from(*sample).abs(); + if i % 2 == 0 { + assert!(*sample >= 0, "positive sample flipped sign: {sample}"); + assert!( + scaled_abs <= i32::from(i16::MAX), + "i16::MAX scaled value wrapped: {sample}" + ); + } else { + assert!(*sample <= 0, "negative sample flipped sign: {sample}"); + assert!( + scaled_abs <= -i32::from(i16::MIN), + "i16::MIN scaled value wrapped: {sample}" + ); + } + } + } + + #[test] + fn test_volume_accessor_returns_clamped_value() { + let samples: Vec = vec![100; 8000]; + let data = raw_pcm_to_base64(&samples); + + let clip = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), Some(5.0)) + .expect("clip should decode with clamped volume"); + assert!((clip.volume() - 1.0).abs() < f32::EPSILON); + } + + #[test] + fn test_invalid_format_string_rejected() { + let samples: Vec = vec![100; 8000]; + let data = raw_pcm_to_base64(&samples); + + let err = decode_loading_clip(&data, Some("mp3"), Some(16_000), Some(1), None) + .expect_err("unknown format should be rejected"); + assert_eq!(err, "loading_audio.format must be \"wav\" or \"pcm\""); + } + + #[test] + fn test_forged_wav_data_chunk_length_rejected() { + // Build a valid small WAV, then overwrite its `data` chunk size field + // (the u32 at the canonical offset 40) with a huge value. `hound` + // parses the header without reading the samples, so without the + // declared-size guard in `decode_wav` this would drive a multi- + // gigabyte `Vec` pre-allocation. The guard must reject it instead. + let samples: Vec = (0..8_000).map(|i| (i % 1000) as i16).collect(); + let mut wav_bytes = { + let spec = hound::WavSpec { + channels: 1, + sample_rate: 16_000, + bits_per_sample: 16, + sample_format: hound::SampleFormat::Int, + }; + let mut cursor = std::io::Cursor::new(Vec::::new()); + { + let mut writer = + hound::WavWriter::new(&mut cursor, spec).expect("WAV writer should be created"); + for sample in &samples { + writer + .write_sample(*sample) + .expect("sample should be written"); + } + writer.finalize().expect("WAV writer should finalize"); + } + cursor.into_inner() + }; + assert!( + wav_bytes.len() > 44, + "a canonical 16-bit PCM WAV has a 44-byte header" + ); + // Forge both the RIFF chunk size (offset 4) and the `data` chunk size + // (offset 40). 0x7FFF_FFF0 bytes of "data" is ~2.1 GB — far past the + // size cap, and even (so `hound` accepts it as whole 2-byte frames). + wav_bytes[4..8].copy_from_slice(&0x7FFF_FFFFu32.to_le_bytes()); + wav_bytes[40..44].copy_from_slice(&0x7FFF_FFF0u32.to_le_bytes()); + + let data = general_purpose::STANDARD.encode(&wav_bytes); + let err = decode_loading_clip(&data, Some("wav"), None, None, None) + .expect_err("a WAV with a forged data-chunk length must be rejected"); + assert!(err.contains("maximum size"), "unexpected error: {err}"); + } +} diff --git a/src/livekit/mod.rs b/src/livekit/mod.rs index 6ef0a6d..3eb7425 100644 --- a/src/livekit/mod.rs +++ b/src/livekit/mod.rs @@ -69,6 +69,7 @@ //! - Supporting the WebSocket API abstraction layer mod client; +pub mod loading_clip; mod manager; pub mod metadata; pub mod operations; @@ -78,6 +79,7 @@ mod types; // Re-export public types and traits pub use client::{AudioCallback, DataCallback, DataMessage, LiveKitClient}; +pub use loading_clip::{LoadingClip, decode_loading_clip}; pub use manager::LiveKitManager; pub use operations::{LiveKitOperation, OperationQueue}; pub use room_handler::LiveKitRoomHandler; From 469d91dea46924af3e8494e6c1b8d066cfd6cb8a Mon Sep 17 00:00:00 2001 From: tigranbs Date: Thu, 21 May 2026 22:03:33 -0700 Subject: [PATCH 2/5] refactor(ws): harden loading-indicator failure handling and tests Follow-up hardening for the loading-indicator audio feature: - Report the original loading_audio decode failure again on a later loading_start, instead of a generic "not available" message; the reason is retained on the connection state. - Give start_loading_audio distinct errors for a missing clip versus a loading track that failed to publish. - Close the reconnect/loading-loop race: tear down the loop and clear the dead loading source together under the loading_loop lock, so a loading_start racing a reconnect can never bind to a stale source. - Allow WAV container overhead in the decoded-payload size guard so a maximum-duration WAV clip is not rejected for its header bytes. - Run the libwebrtc-native loading-audio tests in an isolated, single-threaded CI step; they intermittently segfault under the full unit-test binary's thread concurrency. - Deduplicate the shared loading-clip test helper and add a handler-level test for the missing-clip error path. --- .github/workflows/ci.yml | 11 +++ src/handlers/ws/config_handler.rs | 3 + src/handlers/ws/loading_handler.rs | 103 ++++++++++++++++++++++++++-- src/handlers/ws/state.rs | 7 ++ src/livekit/client/audio.rs | 36 ++++++---- src/livekit/client/connection.rs | 15 +++- src/livekit/client/loading_audio.rs | 64 ++++++++++------- src/livekit/client/mod.rs | 17 +++-- src/livekit/client/tests.rs | 81 +++++++++++++++------- src/livekit/loading_clip.rs | 55 +++++++++++++-- 10 files changed, 310 insertions(+), 82 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 648157c..d2b2d77 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -56,3 +56,14 @@ jobs: cargo build --locked --all-features cargo test --locked --all-features echo "::endgroup::" + + # The `livekit_native_*` tests construct real libwebrtc `NativeAudioSource` + # objects. libwebrtc's global runtime intermittently segfaults under the + # heavy thread concurrency of the full unit-test binary, so those tests are + # `#[ignore]`d out of the run above and executed here in their own isolated, + # single-threaded process instead. + - name: Test LiveKit native audio (isolated) + run: | + echo "::group::Running libwebrtc-native loading-audio tests in isolation" + cargo test --locked --all-features --lib -- --ignored --test-threads=1 livekit_native_ + echo "::endgroup::" diff --git a/src/handlers/ws/config_handler.rs b/src/handlers/ws/config_handler.rs index c0f4799..7c9f5dd 100644 --- a/src/handlers/ws/config_handler.rs +++ b/src/handlers/ws/config_handler.rs @@ -139,6 +139,9 @@ pub async fn handle_config_message( Ok(clip) => Some(clip), Err(message) => { error!("Failed to decode loading_audio: {}", message); + // Retain the reason so a later `loading_start` can + // report it again clearly. + state.write().await.loading_audio_error = Some(message.clone()); let _ = message_tx .send(MessageRoute::Outgoing(OutgoingMessage::Error { message })) .await; diff --git a/src/handlers/ws/loading_handler.rs b/src/handlers/ws/loading_handler.rs index 68d2de1..4cdf121 100644 --- a/src/handlers/ws/loading_handler.rs +++ b/src/handlers/ws/loading_handler.rs @@ -47,16 +47,17 @@ pub async fn handle_loading_start_message( debug!("Processing loading_start command"); // Snapshot the state we need, then drop the read lock immediately. - let (audio_enabled, livekit_client) = { + let (audio_enabled, livekit_client, loading_audio_error) = { let state_guard = state.read().await; ( state_guard.is_audio_enabled(), state_guard.livekit_client.clone(), + state_guard.loading_audio_error.clone(), ) }; // Audio must be enabled. This also covers the "no config received yet" case, - // since audio defaults to disabled. + // since audio defaults to disabled until a config message sets it. if !audio_enabled { send_error( message_tx, @@ -66,6 +67,14 @@ pub async fn handle_loading_start_message( return true; } + // The `loading_audio` supplied in the config message failed to decode. + // The reason was already reported once at config time; report it again + // clearly here rather than a generic "not available". + if let Some(reason) = loading_audio_error { + send_error(message_tx, reason).await; + return true; + } + // A LiveKit room is required to play the loading audio. let livekit_client = match livekit_client { Some(client) => client, @@ -82,13 +91,24 @@ pub async fn handle_loading_start_message( // `start_loading_audio` takes `&self`, so a read lock is sufficient. let client_guard = livekit_client.read().await; if let Err(e) = client_guard.start_loading_audio().await { - let message = match e { - AppError::BadRequest(m) - | AppError::InternalServerError(m) + // Surface the inner message verbatim: `start_loading_audio` already + // produces end-user-facing strings, so the `AppError` `Display` prefix + // ("Bad request: ", etc.) would only add noise for the client. + // + // A `BadRequest` (no clip configured, or the track failed to publish) + // is a normal client-side condition, not a server fault — log it + // quietly; only genuine server faults warrant `error!`. + let (message, is_client_error) = match e { + AppError::BadRequest(m) => (m, true), + AppError::InternalServerError(m) | AppError::NotFound(m) - | AppError::Unauthorized(m) => m, + | AppError::Unauthorized(m) => (m, false), }; - error!("Failed to start loading-indicator audio: {}", message); + if is_client_error { + debug!("loading_start rejected: {message}"); + } else { + error!("Failed to start loading-indicator audio: {message}"); + } send_error(message_tx, message).await; } else { debug!("Loading-indicator audio started"); @@ -135,6 +155,9 @@ mod tests { #[tokio::test] async fn test_handle_loading_start_message_rejects_when_audio_disabled() { + // A fresh ConnectionState has audio disabled, which is also the state + // before any config message — so this covers both the no-config-yet + // case and a config received with audio=false. let state = Arc::new(RwLock::new(ConnectionState::new())); let (message_tx, mut message_rx) = mpsc::channel(4); @@ -150,6 +173,30 @@ mod tests { } } + #[tokio::test] + async fn test_handle_loading_start_message_reports_decode_error() { + // loading_audio was supplied in the config message but failed to + // decode. loading_start must replay the original decode reason, not a + // generic "not available" message. + let mut state_inner = ConnectionState::new(); + state_inner.set_audio_enabled(true); + state_inner.loading_audio_error = + Some("loading_audio.data is not valid base64".to_string()); + let state = Arc::new(RwLock::new(state_inner)); + let (message_tx, mut message_rx) = mpsc::channel(4); + + let continue_processing = handle_loading_start_message(&state, &message_tx).await; + + assert!(continue_processing); + match message_rx.recv().await { + Some(MessageRoute::Outgoing(OutgoingMessage::Error { message })) => { + assert_eq!(message, "loading_audio.data is not valid base64"); + } + Some(_) => panic!("expected the original decode-failure error"), + None => panic!("expected the original decode-failure error"), + } + } + #[tokio::test] async fn test_handle_loading_start_message_rejects_when_no_livekit() { let state = Arc::new(RwLock::new(ConnectionState::new())); @@ -171,6 +218,48 @@ mod tests { } } + #[tokio::test] + async fn test_handle_loading_start_message_forwards_missing_clip_error() { + // Handler-level missing-clip case: audio is enabled and a connected + // LiveKit client is present, but no loading clip was configured. The + // handler must forward `start_loading_audio`'s specific BadRequest + // message verbatim rather than swallow it or send a generic error. + use crate::livekit::{LiveKitClient, LiveKitConfig}; + + let client = LiveKitClient::new(LiveKitConfig { + url: "wss://test-server.com".to_string(), + token: "mock-jwt-token".to_string(), + room_name: "test-room".to_string(), + publish_audio: true, + subscribe_audio: true, + sample_rate: 24_000, + channels: 1, + enable_noise_filter: false, + listen_participants: vec![], + }); + client.set_connected(true).await; + + let mut state_inner = ConnectionState::new(); + state_inner.set_audio_enabled(true); + state_inner.livekit_client = Some(Arc::new(RwLock::new(client))); + let state = Arc::new(RwLock::new(state_inner)); + let (message_tx, mut message_rx) = mpsc::channel(4); + + let continue_processing = handle_loading_start_message(&state, &message_tx).await; + + assert!(continue_processing); + match message_rx.recv().await { + Some(MessageRoute::Outgoing(OutgoingMessage::Error { message })) => { + assert!( + message.contains("no loading audio was configured"), + "unexpected error message: {message}" + ); + } + Some(_) => panic!("expected the missing-clip BadRequest forwarded verbatim"), + None => panic!("expected the missing-clip BadRequest forwarded verbatim"), + } + } + #[tokio::test] async fn test_handle_loading_stop_message_is_silent_noop() { let state = Arc::new(RwLock::new(ConnectionState::new())); diff --git a/src/handlers/ws/state.rs b/src/handlers/ws/state.rs index ba2a5c3..0e90b46 100644 --- a/src/handlers/ws/state.rs +++ b/src/handlers/ws/state.rs @@ -36,6 +36,11 @@ pub struct ConnectionState { pub livekit_local_identity: Option, /// Recording egress ID for cleanup operations pub recording_egress_id: Option, + /// Loading-indicator audio decode failure, if the `loading_audio` supplied + /// in the `config` message failed to decode. Retained so a later + /// `loading_start` can report the original reason instead of a generic + /// "not available" message. + pub loading_audio_error: Option, /// Auth context for this connection (used for room metadata auth_id guard) pub auth: Auth, } @@ -57,6 +62,7 @@ impl ConnectionState { livekit_room_name: None, livekit_local_identity: None, recording_egress_id: None, + loading_audio_error: None, auth: Auth::empty(), } } @@ -72,6 +78,7 @@ impl ConnectionState { livekit_room_name: None, livekit_local_identity: None, recording_egress_id: None, + loading_audio_error: None, auth, } } diff --git a/src/livekit/client/audio.rs b/src/livekit/client/audio.rs index 93cf73e..7f6d9cb 100644 --- a/src/livekit/client/audio.rs +++ b/src/livekit/client/audio.rs @@ -606,25 +606,37 @@ impl LiveKitClient { info!("Successfully reconnected and re-published audio track"); - // Stop any in-progress loading-audio loop: its audio source died with the - // pre-reconnect room. A buffered `NativeAudioSource` keeps accepting frames - // even after its room is gone, so the loop cannot detect the reconnect on - // its own — cancel and abort it here. Clearing `loading_loop` lets a later - // `loading_start` spawn a fresh loop on the re-published track; the client - // must re-send `loading_start` to resume the loading sound. - if let Some(loading) = ctx.loading_loop.lock().await.take() { - loading.token.cancel(); - loading.handle.abort(); + // Tear down any in-progress loading-audio loop and clear the now-dead + // loading source as one step under the `loading_loop` guard. Both this + // teardown and `start_loading_audio` mutate/read `loading_audio_source` + // while holding `loading_loop`, so a `loading_start` racing this reconnect + // can never bind a fresh loop to the stale pre-reconnect source: it either + // runs before this teardown (its loop is aborted here) or after it (it then + // reads the cleared — or freshly re-published — source). Lock order is + // always `loading_loop` then `loading_audio_source`. + // + // A buffered `NativeAudioSource` keeps accepting frames even after its room + // is gone, so the loop cannot detect the reconnect on its own — hence the + // explicit cancel + abort. The client must re-send `loading_start` to + // resume the loading sound after a reconnect. + { + let mut loading_loop_guard = ctx.loading_loop.lock().await; + if let Some(loading) = loading_loop_guard.take() { + loading.token.cancel(); + loading.handle.abort(); + } + *ctx.loading_audio_source.lock().await = None; + *ctx.loading_track_publication.lock().await = None; } // Re-publish the loading-audio track if a loading clip is configured // (non-fatal). The re-published `LocalAudioTrack` handle is intentionally // dropped here: `OperationContext` carries no `loading_audio_track` field, // and the publication alone keeps the track alive — mirroring how the TTS - // track is re-published above. + // track is re-published above. Storing the rebuilt source needs no + // `loading_loop` guard: until it is stored the source reads `None`, so a + // racing `loading_start` simply gets an "unavailable" error and retries. if let Some(clip) = &ctx.loading_clip { - *ctx.loading_audio_source.lock().await = None; - *ctx.loading_track_publication.lock().await = None; match super::loading_audio::publish_loading_audio_track( &local_participant, clip.sample_rate(), diff --git a/src/livekit/client/connection.rs b/src/livekit/client/connection.rs index 63f22eb..445f13e 100644 --- a/src/livekit/client/connection.rs +++ b/src/livekit/client/connection.rs @@ -92,7 +92,13 @@ impl LiveKitClient { let _ = tokio::time::timeout(std::time::Duration::from_millis(500), handle).await; } - // Tear down the loading-audio loop: cancel, await with timeout, abort as backstop. + // Tear down the loading-audio loop: cancel, await with timeout, abort as + // backstop. Unlike `process_reconnect`, `disconnect` does not co-hold the + // `loading_loop` and `loading_audio_source` locks: its `&mut self` receiver + // gives it exclusive access to the client (no concurrent `&self` + // `start_loading_audio` / `stop_loading_audio`), and the operation worker + // was already shut down above — so nothing can spawn or rebind a loop while + // these fields are torn down. if let Some(loading) = self.loading_loop.lock().await.take() { loading.token.cancel(); let abort = loading.handle.abort_handle(); @@ -114,8 +120,11 @@ impl LiveKitClient { *self.local_track_publication.lock().await = None; self.has_audio_source_atomic.store(false, Ordering::Release); - // Clear loading-audio resources only after the loop has been awaited, - // so no stray fade-out frame is captured into a half-torn-down source. + // Clear the loading-audio resources only after the loop task has been + // awaited/aborted above. The loop owns its own `Arc` clone of the + // source, so clearing these fields never frees it mid-capture — but + // awaiting first keeps teardown deterministic: the loop has provably + // stopped before its source/track/publication are dropped here. *self.loading_audio_source.lock().await = None; self.loading_audio_track = None; *self.loading_track_publication.lock().await = None; diff --git a/src/livekit/client/loading_audio.rs b/src/livekit/client/loading_audio.rs index 25c6256..d8d71b4 100644 --- a/src/livekit/client/loading_audio.rs +++ b/src/livekit/client/loading_audio.rs @@ -240,29 +240,25 @@ impl LiveKitClient { )); } - let source = { - let guard = self.loading_audio_source.lock().await; - guard.as_ref().map(Arc::clone) - }; - let source = match source { - Some(source) => source, - None => { - return Err(AppError::BadRequest( - "loading audio is not available for this session".to_string(), - )); - } - }; - + // No loading clip was configured for this session. let clip = match &self.loading_clip { Some(clip) => Arc::clone(clip), None => { return Err(AppError::BadRequest( - "loading audio is not available for this session".to_string(), + "no loading audio was configured for this session".to_string(), )); } }; + // Hold `loading_loop` across the whole start, and read + // `loading_audio_source` while still holding it. A concurrent reconnect + // (`process_reconnect`) tears down the loop and clears the loading + // source under this same `loading_loop` guard, so taking the lock first + // guarantees this loop binds to the current source — never a stale, + // dead pre-reconnect one. Lock order is always `loading_loop` then + // `loading_audio_source`. let mut loop_guard = self.loading_loop.lock().await; + // A finished handle is stale and is replaced; a live one means the loop // is already running, so starting again is an idempotent no-op. if let Some(existing) = loop_guard.as_ref() @@ -272,6 +268,22 @@ impl LiveKitClient { return Ok(()); } + // A clip was configured but the dedicated "loading-audio" track failed + // to publish (or a reconnect is mid-rebuild), so there is no source to + // feed frames into. + let source = { + let guard = self.loading_audio_source.lock().await; + guard.as_ref().map(Arc::clone) + }; + let source = match source { + Some(source) => source, + None => { + return Err(AppError::BadRequest( + "the loading audio track is unavailable; it failed to publish".to_string(), + )); + } + }; + let token = CancellationToken::new(); // Isolation invariant: the spawned loop owns only its dedicated audio // source, the loading clip, and its cancellation token. It never @@ -321,7 +333,6 @@ async fn run_loading_loop( ) { let sample_rate = clip.sample_rate(); let channels = clip.channels(); - let samples_per_channel = (sample_rate / 100) as usize; let frame_len = loading_frame_len(sample_rate, channels); if frame_len == 0 || clip.samples().is_empty() { @@ -329,6 +340,11 @@ async fn run_loading_loop( return; } + // Per-channel sample count for one 10 ms frame. Derived from `frame_len` + // (the all-channels length) so `loading_frame_len` stays the single source + // of truth for the 10 ms math. `channels >= 1` here because `frame_len != 0`. + let samples_per_channel = frame_len / channels as usize; + let mut frame_buf: Vec = vec![0; frame_len]; let mut cursor: usize = 0; @@ -521,15 +537,17 @@ mod tests { } } + // Constructs a real libwebrtc `NativeAudioSource`. libwebrtc's global + // runtime intermittently segfaults under the full unit-test binary's + // concurrency, so this test is `#[ignore]`d from the default run and + // executed isolated by a dedicated CI step (see `.github/workflows/ci.yml`). + // Run locally with: + // cargo test --all-features --lib -- --ignored --test-threads=1 livekit_native_ #[tokio::test] - async fn test_run_loading_loop_exits_cleanly_when_precancelled() { - // A 500 ms 16 kHz mono clip, well above the minimum duration. - let samples: Vec = (0..8_000).map(|i| (i % 1000) as i16).collect(); - let data = crate::livekit::loading_clip::raw_pcm_to_base64(&samples); - let clip = Arc::new( - crate::livekit::decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) - .expect("raw PCM clip should decode"), - ); + #[ignore = "creates native libwebrtc objects; run isolated via the dedicated CI step (see ci.yml)"] + async fn livekit_native_run_loop_exits_when_precancelled() { + // The canonical 500 ms 16 kHz mono test clip, well above the minimum duration. + let clip = Arc::new(crate::livekit::loading_clip::make_test_loading_clip()); let source = Arc::new(NativeAudioSource::new( AudioSourceOptions { diff --git a/src/livekit/client/mod.rs b/src/livekit/client/mod.rs index b7163f2..d5b579e 100644 --- a/src/livekit/client/mod.rs +++ b/src/livekit/client/mod.rs @@ -290,13 +290,16 @@ impl Drop for LiveKitClient { warn!("LiveKitClient dropped without explicit disconnect call"); } - // Backstop for the loading-audio loop. `disconnect` is the normal - // teardown path, but session cleanup may skip it if the LiveKit write - // lock cannot be acquired in time. A buffered `NativeAudioSource` keeps - // accepting frames even after the room is gone, so the loop would never - // observe a `capture_frame` error and never self-terminate. Cancelling - // the token and aborting the task here keeps the loop from being - // orphaned regardless of how the client is dropped. + // Best-effort backstop for the loading-audio loop. `disconnect` is the + // normal teardown path, but session cleanup may skip it if the LiveKit + // write lock cannot be acquired in time. A buffered `NativeAudioSource` + // keeps accepting frames even after the room is gone, so the loop would + // never observe a `capture_frame` error and never self-terminate. + // Cancelling the token and aborting the task here reaps the loop on the + // common drop paths. `try_lock` is used because `Drop` cannot block; if + // it loses a momentary race for the `loading_loop` lock, the loop is + // left for that lock holder's own teardown (`disconnect` / reconnect) to + // reap. if let Ok(mut guard) = self.loading_loop.try_lock() && let Some(loading) = guard.take() { diff --git a/src/livekit/client/tests.rs b/src/livekit/client/tests.rs index 8c6e90d..0608ded 100644 --- a/src/livekit/client/tests.rs +++ b/src/livekit/client/tests.rs @@ -319,13 +319,6 @@ async fn test_livekit_client_operation_priority_ordering() { assert!(queue.is_none()); // Queue only exists after connect } -/// Builds a base64-encoded raw 16-bit PCM clip long enough to pass the -/// minimum-duration validation (500 ms of 16 kHz mono audio). -fn make_loading_clip_base64() -> String { - let samples: Vec = (0..8_000).map(|i| (i % 1000) as i16).collect(); - crate::livekit::loading_clip::raw_pcm_to_base64(&samples) -} - #[tokio::test] async fn test_livekit_client_loading_fields_default() { let config = create_test_config(); @@ -340,9 +333,7 @@ async fn test_livekit_client_loading_fields_default() { #[tokio::test] async fn test_set_loading_audio_clip_stores_clip() { - let data = make_loading_clip_base64(); - let clip = crate::livekit::decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) - .expect("raw PCM loading clip should decode"); + let clip = crate::livekit::loading_clip::make_test_loading_clip(); let config = create_test_config(); let mut client = LiveKitClient::new(config); @@ -370,9 +361,7 @@ async fn test_setup_loading_audio_track_without_clip_is_noop() { #[tokio::test] async fn test_setup_loading_audio_track_with_clip_requires_room() { - let data = make_loading_clip_base64(); - let clip = crate::livekit::decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) - .expect("raw PCM loading clip should decode"); + let clip = crate::livekit::loading_clip::make_test_loading_clip(); let mut client = LiveKitClient::new(create_test_config()); client.set_loading_audio_clip(clip); @@ -417,11 +406,43 @@ async fn test_start_loading_audio_connected_without_clip_errors() { let client = LiveKitClient::new(config); client.set_connected(true).await; - let result = client.start_loading_audio().await; - assert!( - result.is_err(), - "start_loading_audio should fail without a loading source/clip" - ); + // Connected, but no loading clip was ever configured. The error must name + // that specific cause rather than a generic message. + let err = client + .start_loading_audio() + .await + .expect_err("start_loading_audio should fail without a loading clip"); + match err { + AppError::BadRequest(msg) => assert!( + msg.contains("no loading audio was configured"), + "unexpected error message: {msg}" + ), + other => panic!("expected BadRequest for the missing-clip case, got {other:?}"), + } +} + +#[tokio::test] +async fn test_start_loading_audio_with_clip_but_no_source_errors() { + // A clip is configured but the dedicated loading track never published, so + // there is no audio source. start_loading_audio must report that specific + // cause, distinct from the missing-clip message. + let clip = crate::livekit::loading_clip::make_test_loading_clip(); + + let mut client = LiveKitClient::new(create_test_config()); + client.set_connected(true).await; + client.set_loading_audio_clip(clip); + + let err = client + .start_loading_audio() + .await + .expect_err("start_loading_audio should fail with a clip but no published source"); + match err { + AppError::BadRequest(msg) => assert!( + msg.contains("failed to publish"), + "unexpected error message: {msg}" + ), + other => panic!("expected BadRequest for the missing-source case, got {other:?}"), + } } #[tokio::test] @@ -439,8 +460,17 @@ async fn test_disconnect_clears_loading_fields() { assert!(!client.has_loading_loop().await); } +// NOTE: the `livekit_native_*` tests below construct real libwebrtc +// `NativeAudioSource` objects. libwebrtc's lazily-initialised global runtime is +// not robust against the heavy thread concurrency of the full unit-test binary +// and intermittently segfaults when these run alongside the rest of the suite. +// They are therefore `#[ignore]`d out of the default `cargo test` run and +// executed isolated (their own process) by a dedicated CI step — see +// `.github/workflows/ci.yml`. Run locally with: +// cargo test --all-features --lib -- --ignored --test-threads=1 livekit_native_ #[tokio::test] -async fn test_two_independent_native_audio_sources() { +#[ignore = "creates native libwebrtc objects; run isolated via the dedicated CI step (see ci.yml)"] +async fn livekit_native_two_independent_audio_sources() { use livekit::track::LocalAudioTrack; use livekit::webrtc::audio_source::native::NativeAudioSource; use livekit::webrtc::prelude::{AudioSourceOptions, RtcAudioSource}; @@ -483,9 +513,7 @@ async fn connected_client_with_loading_source() -> LiveKitClient { use livekit::webrtc::audio_source::native::NativeAudioSource; use livekit::webrtc::prelude::AudioSourceOptions; - let data = make_loading_clip_base64(); - let clip = crate::livekit::decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) - .expect("raw PCM loading clip should decode"); + let clip = crate::livekit::loading_clip::make_test_loading_clip(); let mut client = LiveKitClient::new(create_test_config()); client.set_connected(true).await; @@ -506,7 +534,8 @@ async fn connected_client_with_loading_source() -> LiveKitClient { } #[tokio::test] -async fn test_start_loading_audio_idempotent_and_disconnect_tears_down() { +#[ignore = "creates native libwebrtc objects; run isolated via the dedicated CI step (see ci.yml)"] +async fn livekit_native_start_idempotent_and_disconnect_teardown() { let mut client = connected_client_with_loading_source().await; // The first start spawns the loop. @@ -535,7 +564,8 @@ async fn test_start_loading_audio_idempotent_and_disconnect_tears_down() { } #[tokio::test] -async fn test_rapid_start_stop_loading_audio_is_clean() { +#[ignore = "creates native libwebrtc objects; run isolated via the dedicated CI step (see ci.yml)"] +async fn livekit_native_rapid_start_stop_is_clean() { let mut client = connected_client_with_loading_source().await; // Rapidly alternating start/stop must never panic, and a disconnect @@ -560,7 +590,8 @@ async fn test_rapid_start_stop_loading_audio_is_clean() { } #[tokio::test] -async fn test_drop_cancels_active_loading_loop() { +#[ignore = "creates native libwebrtc objects; run isolated via the dedicated CI step (see ci.yml)"] +async fn livekit_native_drop_cancels_active_loop() { let client = connected_client_with_loading_source().await; client .start_loading_audio() diff --git a/src/livekit/loading_clip.rs b/src/livekit/loading_clip.rs index fa6f697..f645694 100644 --- a/src/livekit/loading_clip.rs +++ b/src/livekit/loading_clip.rs @@ -42,6 +42,18 @@ pub const MAX_LOADING_AUDIO_DECODED_BYTES: usize = (LOADING_AUDIO_SAMPLE_RATE_MA * (MAX_LOADING_AUDIO_MS as usize) / 1000; +/// Maximum accepted size of the base64-decoded payload, in bytes. +/// +/// This is [`MAX_LOADING_AUDIO_DECODED_BYTES`] (the worst-case raw PCM size) +/// plus a fixed allowance for WAV container overhead. A canonical WAV header is +/// 44 bytes, but real-world files carry extra metadata chunks (`LIST`, `fact`, +/// …); the allowance keeps a maximum-duration WAV clip from being rejected +/// purely for its header. The decoded *audio* length is still bounded +/// authoritatively by the duration check, so the slack cannot admit an +/// over-long clip — it only stops the wire-payload guard from being stricter +/// than the audio guard. +pub const MAX_LOADING_AUDIO_PAYLOAD_BYTES: usize = MAX_LOADING_AUDIO_DECODED_BYTES + 4_096; + /// A decoded, validated, ready-to-play loading audio sample. /// /// The PCM data in [`LoadingClip::samples`] is flat and interleaved when @@ -128,10 +140,10 @@ pub fn decode_loading_clip( // encoded characters), so an obviously oversized payload is rejected // before allocating the decoded buffer. let estimated_decoded_bytes = data.len() / 4 * 3; - if estimated_decoded_bytes > MAX_LOADING_AUDIO_DECODED_BYTES { + if estimated_decoded_bytes > MAX_LOADING_AUDIO_PAYLOAD_BYTES { return Err(format!( "loading_audio.data decoded audio would exceed the maximum size of \ - {MAX_LOADING_AUDIO_DECODED_BYTES} bytes" + {MAX_LOADING_AUDIO_PAYLOAD_BYTES} bytes" )); } @@ -140,11 +152,14 @@ pub fn decode_loading_clip( .decode(data) .map_err(|_| "loading_audio.data is not valid base64".to_string())?; - // 3. Post-decode size guard against the exact decoded length. - if bytes.len() > MAX_LOADING_AUDIO_DECODED_BYTES { + // 3. Post-decode size guard against the exact decoded payload length. For + // WAV this includes the container header, hence the payload bound + // rather than the raw-PCM bound; the duration check below still bounds + // the audio itself. + if bytes.len() > MAX_LOADING_AUDIO_PAYLOAD_BYTES { return Err(format!( "loading_audio.data decoded audio exceeds the maximum size of \ - {MAX_LOADING_AUDIO_DECODED_BYTES} bytes" + {MAX_LOADING_AUDIO_PAYLOAD_BYTES} bytes" )); } @@ -297,6 +312,19 @@ pub(crate) fn raw_pcm_to_base64(samples: &[i16]) -> String { general_purpose::STANDARD.encode(bytes) } +/// Decodes the canonical test loading clip: 500 ms of 16 kHz mono raw PCM, +/// comfortably above the minimum-duration validation bound. +/// +/// Test-only helper shared by the loading-audio test suites in this crate so +/// the `raw_pcm_to_base64` + `decode_loading_clip` idiom is written once. +#[cfg(test)] +pub(crate) fn make_test_loading_clip() -> LoadingClip { + let samples: Vec = (0..8_000).map(|i| (i % 1000) as i16).collect(); + let data = raw_pcm_to_base64(&samples); + decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(1), None) + .expect("the canonical test loading clip must decode") +} + #[cfg(test)] mod tests { use super::*; @@ -490,6 +518,23 @@ mod tests { assert!(err.contains("maximum size"), "unexpected error: {err}"); } + #[test] + fn test_max_duration_wav_is_accepted() { + // A maximum-spec WAV — 10 s, 48 kHz, stereo, 16-bit — carries exactly + // MAX_LOADING_AUDIO_DECODED_BYTES of PCM *plus* a WAV header. The + // payload guard must allow the container overhead so this valid clip is + // not rejected purely for its header bytes. + let frames = (LOADING_AUDIO_SAMPLE_RATE_MAX * MAX_LOADING_AUDIO_MS / 1000) as usize; + let samples: Vec = vec![0; frames * 2]; // stereo: 2 samples per frame + let data = make_wav_base64(LOADING_AUDIO_SAMPLE_RATE_MAX, 2, &samples); + + let clip = decode_loading_clip(&data, None, None, None, None) + .expect("a maximum-duration WAV clip must be accepted"); + assert_eq!(clip.channels(), 2); + assert_eq!(clip.sample_rate(), LOADING_AUDIO_SAMPLE_RATE_MAX); + assert_eq!(clip.samples().len(), frames * 2); + } + #[test] fn test_too_short_clip_rejected() { // 1000 samples at 16 kHz mono == ~62 ms, below MIN_LOADING_AUDIO_MS. From 6710d1e210974697f79f764af1fda91d87d7ff97 Mon Sep 17 00:00:00 2001 From: tigranbs Date: Sun, 24 May 2026 22:39:01 -0700 Subject: [PATCH 3/5] fix(loading): harden loop lifecycle, errors, and tests Clear loading_loop when the playback task exits; fix stop cursor continuity and stale decode errors; use BadRequest when LiveKit is not connected; honor cancel during fade-out; add tests and docs; ignore local loading-indicator.md planning file. --- .gitignore | 3 + CHANGELOG.md | 6 ++ docs/livekit_integration.md | 4 +- docs/websocket.md | 3 + src/handlers/ws/config_handler.rs | 5 +- src/handlers/ws/loading_handler.rs | 146 ++++++++++++++++++++++++++-- src/livekit/client/loading_audio.rs | 99 ++++++++++++++----- src/livekit/client/mod.rs | 6 +- src/livekit/client/tests.rs | 67 ++++++++++++- src/livekit/loading_clip.rs | 21 +++- 10 files changed, 316 insertions(+), 44 deletions(-) diff --git a/.gitignore b/.gitignore index b105c56..ab229d4 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,6 @@ libonnxruntime.so onnxruntime/ dist/ .mcgravity/ + +# Local planning doc for loading-indicator feature (not part of the product tree) +loading-indicator.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 3dde65c..1fb04c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ All notable changes to Sayna will be documented in this file. +## [Unreleased] + +### Features + +- **websocket:** Add loading indicator audio (`loading_audio` in `config`, `loading_start` / `loading_stop` commands) on a dedicated `"loading-audio"` LiveKit track, independent of TTS + ## [0.1.15] - 2026-04-30 ### Features diff --git a/docs/livekit_integration.md b/docs/livekit_integration.md index f753f10..a31c88d 100644 --- a/docs/livekit_integration.md +++ b/docs/livekit_integration.md @@ -94,9 +94,9 @@ Sayna provides comprehensive LiveKit integration with automatic SIP infrastructu ## Published Audio Tracks -When a WebSocket session is audio-enabled and joins a LiveKit room, the Sayna agent participant publishes synthesized speech on a single audio track named `"tts-audio"`. +When a WebSocket session is audio-enabled and joins a LiveKit room, the Sayna agent participant publishes synthesized speech on an audio track named `"tts-audio"`. -If that session's `config` message also includes a `loading_audio` object (see [Loading Indicator](websocket.md#loading-indicator) in the WebSocket reference), the agent participant publishes a **second** audio track named `"loading-audio"`, carrying the loading indicator sound. In that case the agent publishes **two** audio tracks: +If that session's `config` message also includes a `loading_audio` object (see [Loading Indicator](websocket.md#loading-indicator) in the WebSocket reference), the agent participant also publishes a **second** audio track named `"loading-audio"`, carrying the loading indicator sound. The agent then publishes **two** audio tracks: | Track name | Carries | |------------|---------| diff --git a/docs/websocket.md b/docs/websocket.md index a5c9aca..67d477b 100644 --- a/docs/websocket.md +++ b/docs/websocket.md @@ -1559,6 +1559,8 @@ To use the loading indicator, include a `loading_audio` object in the `config` m - Decoded duration must be between ~250 ms and ~10 s (**1–4 s recommended**). - Decoded size is capped (~1.9 MB). +**WebSocket message size:** A `config` message with a large `loading_audio.data` blob is still a single text frame. Sayna uses the default Axum/tungstenite maximum message size, which is well above the decoded-size cap above; clips within the documented limits do not require a client-side frame-size change. + **WAV vs. raw PCM:** - **WAV** — supply a complete 16-bit integer WAV file. The header is authoritative, so `sample_rate` and `channels` in the config are ignored. @@ -1578,6 +1580,7 @@ The loop is controlled with the [`loading_start`](#5-loading-start-message) and - `loading_stop` stops the loop with a short fade-out (idempotent and always silent). - The loop is controlled **exclusively** by these two commands. The `speak` and `clear` commands do **not** affect it — they act only on the speech (STT/TTS) pipeline. - The loop also stops automatically on session teardown / disconnect. +- The server does **not** stop the loop when VAD detects user speech. If you want loading audio to end on barge-in, send `loading_stop` when you handle a `vad_event` or `stt_result`. **Required client guidance:** Because `speak` does not stop the loop, and the loading track and TTS track are independent and can be audible at the same time, **send `loading_stop` before (or together with) `speak`** in the normal "thinking finished, now answer" flow. Otherwise the loading sound is heard underneath the spoken answer. diff --git a/src/handlers/ws/config_handler.rs b/src/handlers/ws/config_handler.rs index 7c9f5dd..fb2be0b 100644 --- a/src/handlers/ws/config_handler.rs +++ b/src/handlers/ws/config_handler.rs @@ -107,11 +107,14 @@ pub async fn handle_config_message( return true; } - // Store audio_enabled flag in connection state + // Store audio_enabled flag in connection state. Reset any prior + // loading-audio decode error so the field reflects only this config and a + // later `loading_start` cannot replay a stale failure. { let mut state_guard = state.write().await; state_guard.set_audio_enabled(audio_enabled); state_guard.stream_id = Some(stream_id.clone()); + state_guard.loading_audio_error = None; } debug!(stream_id = %stream_id, "Stored stream_id in connection state"); diff --git a/src/handlers/ws/loading_handler.rs b/src/handlers/ws/loading_handler.rs index 4cdf121..ce20667 100644 --- a/src/handlers/ws/loading_handler.rs +++ b/src/handlers/ws/loading_handler.rs @@ -47,21 +47,29 @@ pub async fn handle_loading_start_message( debug!("Processing loading_start command"); // Snapshot the state we need, then drop the read lock immediately. - let (audio_enabled, livekit_client, loading_audio_error) = { + let (stream_id, audio_enabled, livekit_client, loading_audio_error) = { let state_guard = state.read().await; ( + state_guard.stream_id.clone(), state_guard.is_audio_enabled(), state_guard.livekit_client.clone(), state_guard.loading_audio_error.clone(), ) }; - // Audio must be enabled. This also covers the "no config received yet" case, - // since audio defaults to disabled until a config message sets it. + if stream_id.is_none() { + send_error( + message_tx, + "Send a config message first before using loading_start.", + ) + .await; + return true; + } + if !audio_enabled { send_error( message_tx, - "Loading indicator requires audio to be enabled. Send a config message with audio=true first.", + "Loading indicator requires audio to be enabled. Send a config message with audio=true.", ) .await; return true; @@ -155,10 +163,9 @@ mod tests { #[tokio::test] async fn test_handle_loading_start_message_rejects_when_audio_disabled() { - // A fresh ConnectionState has audio disabled, which is also the state - // before any config message — so this covers both the no-config-yet - // case and a config received with audio=false. - let state = Arc::new(RwLock::new(ConnectionState::new())); + let mut state_inner = ConnectionState::new(); + state_inner.stream_id = Some("test-stream".to_string()); + let state = Arc::new(RwLock::new(state_inner)); let (message_tx, mut message_rx) = mpsc::channel(4); let continue_processing = handle_loading_start_message(&state, &message_tx).await; @@ -179,6 +186,7 @@ mod tests { // decode. loading_start must replay the original decode reason, not a // generic "not available" message. let mut state_inner = ConnectionState::new(); + state_inner.stream_id = Some("test-stream".to_string()); state_inner.set_audio_enabled(true); state_inner.loading_audio_error = Some("loading_audio.data is not valid base64".to_string()); @@ -201,7 +209,8 @@ mod tests { async fn test_handle_loading_start_message_rejects_when_no_livekit() { let state = Arc::new(RwLock::new(ConnectionState::new())); { - let state_guard = state.write().await; + let mut state_guard = state.write().await; + state_guard.stream_id = Some("test-stream".to_string()); state_guard.set_audio_enabled(true); } let (message_tx, mut message_rx) = mpsc::channel(4); @@ -240,6 +249,7 @@ mod tests { client.set_connected(true).await; let mut state_inner = ConnectionState::new(); + state_inner.stream_id = Some("test-stream".to_string()); state_inner.set_audio_enabled(true); state_inner.livekit_client = Some(Arc::new(RwLock::new(client))); let state = Arc::new(RwLock::new(state_inner)); @@ -251,7 +261,7 @@ mod tests { match message_rx.recv().await { Some(MessageRoute::Outgoing(OutgoingMessage::Error { message })) => { assert!( - message.contains("no loading audio was configured"), + message.contains("no loading audio configured"), "unexpected error message: {message}" ); } @@ -260,6 +270,122 @@ mod tests { } } + #[tokio::test] + async fn test_handle_loading_start_message_rejects_before_config() { + let state = Arc::new(RwLock::new(ConnectionState::new())); + let (message_tx, mut message_rx) = mpsc::channel(4); + + let continue_processing = handle_loading_start_message(&state, &message_tx).await; + + assert!(continue_processing); + match message_rx.recv().await { + Some(MessageRoute::Outgoing(OutgoingMessage::Error { message })) => { + assert!( + message.contains("config message first"), + "unexpected error: {message}" + ); + } + Some(_) => panic!("expected config-first error"), + None => panic!("expected config-first error"), + } + } + + #[tokio::test] + async fn test_handle_loading_stop_message_silent_with_client_no_loop() { + use crate::livekit::{LiveKitClient, LiveKitConfig}; + + let client = LiveKitClient::new(LiveKitConfig { + url: "wss://test-server.com".to_string(), + token: "mock-jwt-token".to_string(), + room_name: "test-room".to_string(), + publish_audio: true, + subscribe_audio: true, + sample_rate: 24_000, + channels: 1, + enable_noise_filter: false, + listen_participants: vec![], + }); + client.set_connected(true).await; + + let mut state_inner = ConnectionState::new(); + state_inner.set_audio_enabled(true); + state_inner.stream_id = Some("test-stream".to_string()); + state_inner.livekit_client = Some(Arc::new(RwLock::new(client))); + let state = Arc::new(RwLock::new(state_inner)); + let (message_tx, mut message_rx): ( + mpsc::Sender, + mpsc::Receiver, + ) = mpsc::channel(4); + + let continue_processing = handle_loading_stop_message(&state).await; + drop(message_tx); + + assert!(continue_processing); + assert!( + message_rx.try_recv().is_err(), + "loading_stop must be silent when a LiveKit client exists but no loop is running" + ); + } + + #[tokio::test] + async fn test_handle_loading_start_message_success_is_silent() { + use crate::livekit::loading_clip::make_test_loading_clip; + use crate::livekit::{LiveKitClient, LiveKitConfig}; + use livekit::webrtc::audio_source::native::NativeAudioSource; + use livekit::webrtc::prelude::AudioSourceOptions; + + let clip = make_test_loading_clip(); + let mut client = LiveKitClient::new(LiveKitConfig { + url: "wss://test-server.com".to_string(), + token: "mock-jwt-token".to_string(), + room_name: "test-room".to_string(), + publish_audio: true, + subscribe_audio: true, + sample_rate: 24_000, + channels: 1, + enable_noise_filter: false, + listen_participants: vec![], + }); + client.set_connected(true).await; + client.set_loading_audio_clip(clip); + + let source = Arc::new(NativeAudioSource::new( + AudioSourceOptions { + echo_cancellation: false, + noise_suppression: false, + auto_gain_control: false, + }, + 16_000, + 1, + 100, // NativeAudioSource queue depth in ms (matches LOADING_AUDIO_QUEUE_SIZE_MS) + )); + *client.loading_audio_source.lock().await = Some(source); + + let mut state_inner = ConnectionState::new(); + state_inner.set_audio_enabled(true); + state_inner.stream_id = Some("test-stream".to_string()); + state_inner.livekit_client = Some(Arc::new(RwLock::new(client))); + let state = Arc::new(RwLock::new(state_inner)); + let (message_tx, mut message_rx) = mpsc::channel(4); + + let continue_processing = handle_loading_start_message(&state, &message_tx).await; + drop(message_tx); + + assert!(continue_processing); + assert!( + message_rx.try_recv().is_err(), + "successful loading_start must not emit any WebSocket message" + ); + + // Tear down the spawned loop so the test does not leak a background task. + { + let guard = state.read().await; + if let Some(lk) = &guard.livekit_client { + lk.read().await.stop_loading_audio().await; + } + } + } + #[tokio::test] async fn test_handle_loading_stop_message_is_silent_noop() { let state = Arc::new(RwLock::new(ConnectionState::new())); diff --git a/src/livekit/client/loading_audio.rs b/src/livekit/client/loading_audio.rs index d8d71b4..c50a43d 100644 --- a/src/livekit/client/loading_audio.rs +++ b/src/livekit/client/loading_audio.rs @@ -20,6 +20,7 @@ use livekit::prelude::LocalTrackPublication; use livekit::track::{LocalAudioTrack, LocalTrack, TrackSource}; use livekit::webrtc::audio_source::native::NativeAudioSource; use livekit::webrtc::prelude::{AudioFrame, AudioSourceOptions, RtcAudioSource}; +use tokio::sync::Mutex; use tokio_util::sync::CancellationToken; use tracing::{debug, info, warn}; @@ -45,7 +46,7 @@ const LOADING_AUDIO_FADE_OUT_MS: u32 = 30; /// mirrors LiveKit's `BackgroundAudioPlayer` reference design: once the queue /// fills, `capture_frame().await` applies real-time back-pressure, pacing the /// loop at one 10 ms frame per 10 ms of wall-clock with no manual timer. -pub(super) const LOADING_AUDIO_QUEUE_SIZE_MS: u32 = 100; +pub(crate) const LOADING_AUDIO_QUEUE_SIZE_MS: u32 = 100; /// Handle to a running loading-audio loop: its task handle and cancellation token. /// @@ -121,6 +122,22 @@ pub(super) fn apply_fade(frame: &mut [i16], start_index: usize, total: usize) -> index } +/// Builds one 10 ms `AudioFrame` borrowing `frame_buf`, tagged with the loop's +/// fixed `sample_rate`, channel count, and per-channel sample count. +fn loading_audio_frame( + frame_buf: &[i16], + sample_rate: u32, + channels: u16, + samples_per_channel: usize, +) -> AudioFrame<'_> { + AudioFrame { + data: std::borrow::Cow::Borrowed(frame_buf), + sample_rate, + num_channels: channels as u32, + samples_per_channel: samples_per_channel as u32, + } +} + /// Creates the dedicated loading audio source plus the "loading-audio" track /// and publishes it on the given participant. /// @@ -235,8 +252,9 @@ impl LiveKitClient { /// clip / source is available for this session. pub async fn start_loading_audio(&self) -> Result<(), AppError> { if !self.is_connected() { - return Err(AppError::InternalServerError( - "Not connected to LiveKit room".to_string(), + return Err(AppError::BadRequest( + "loading indicator requires an active LiveKit connection; connect to a room first" + .to_string(), )); } @@ -245,11 +263,18 @@ impl LiveKitClient { Some(clip) => Arc::clone(clip), None => { return Err(AppError::BadRequest( - "no loading audio was configured for this session".to_string(), + "no loading audio configured".to_string(), )); } }; + if loading_frame_len(clip.sample_rate(), clip.channels()) == 0 || clip.samples().is_empty() + { + return Err(AppError::BadRequest( + "loading audio clip has no playable frames".to_string(), + )); + } + // Hold `loading_loop` across the whole start, and read // `loading_audio_source` while still holding it. A concurrent reconnect // (`process_reconnect`) tears down the loop and clears the loading @@ -285,11 +310,17 @@ impl LiveKitClient { }; let token = CancellationToken::new(); + let loading_loop_slot = Arc::clone(&self.loading_loop); // Isolation invariant: the spawned loop owns only its dedicated audio // source, the loading clip, and its cancellation token. It never // touches the TTS operation_queue, audio_queue, audio_generation, or // has_audio_source_atomic. - let handle = tokio::spawn(run_loading_loop(source, clip, token.clone())); + let handle = tokio::spawn(run_loading_loop( + source, + clip, + token.clone(), + loading_loop_slot, + )); *loop_guard = Some(LoadingLoopHandle { handle, token }); info!("Started loading audio loop"); @@ -299,8 +330,8 @@ impl LiveKitClient { /// Stops a running loading-audio loop, triggering its short fade-out. /// /// This is infallible, silent, and idempotent: with no loop running it does - /// nothing. The loop's handle is intentionally left in place so that - /// disconnect (or the next start) can await or replace it. + /// nothing. The loop task clears the `loading_loop` slot when it exits; + /// disconnect can still cancel and await the handle as a backstop. pub async fn stop_loading_audio(&self) { let loop_guard = self.loading_loop.lock().await; if let Some(existing) = loop_guard.as_ref() { @@ -321,6 +352,18 @@ impl LiveKitClient { } } +/// Clears the `loading_loop` slot when the loop task exits (normal stop, fade +/// complete, or early error). Uses `try_lock` so `Drop` never blocks. +struct ClearLoadingLoopOnExit(Arc>>); + +impl Drop for ClearLoadingLoopOnExit { + fn drop(&mut self) { + if let Ok(mut guard) = self.0.try_lock() { + guard.take(); + } + } +} + /// Runs the loading-audio playback loop until cancelled, then fades out. /// /// The task holds only its `Arc`, `Arc`, and @@ -330,13 +373,15 @@ async fn run_loading_loop( source: Arc, clip: Arc, token: CancellationToken, + loading_loop: Arc>>, ) { + let _clear_slot = ClearLoadingLoopOnExit(loading_loop); let sample_rate = clip.sample_rate(); let channels = clip.channels(); let frame_len = loading_frame_len(sample_rate, channels); if frame_len == 0 || clip.samples().is_empty() { - warn!("Loading audio clip has no playable frames; loop not started"); + warn!("Loading audio clip has no playable frames; loop exiting immediately"); return; } @@ -353,13 +398,13 @@ async fn run_loading_loop( // back-pressure, pacing the loop at one 10 ms frame per 10 ms of wall-clock // — so no manual timer is needed. loop { - cursor = fill_loading_frame(clip.samples(), cursor, &mut frame_buf); - let frame = AudioFrame { - data: std::borrow::Cow::Borrowed(&frame_buf), - sample_rate, - num_channels: channels as u32, - samples_per_channel: samples_per_channel as u32, - }; + // Fill the next frame, but commit the cursor advance only once the + // frame has actually been captured. If the cancellation token fires + // first, `cursor` still points at the start of this uncaptured frame, + // so the fade-out below resumes exactly where the last captured frame + // ended — no skipped frame and no discontinuity at the stop point. + let next_cursor = fill_loading_frame(clip.samples(), cursor, &mut frame_buf); + let frame = loading_audio_frame(&frame_buf, sample_rate, channels, samples_per_channel); tokio::select! { biased; @@ -369,6 +414,7 @@ async fn run_loading_loop( debug!("Loading audio source closed; loop exiting without fade-out"); return; } + cursor = next_cursor; } } } @@ -381,16 +427,20 @@ async fn run_loading_loop( let mut fade_index: usize = 0; for _ in 0..fade_frames { + if token.is_cancelled() { + break; + } cursor = fill_loading_frame(clip.samples(), cursor, &mut frame_buf); fade_index = apply_fade(&mut frame_buf, fade_index, total_fade_samples); - let frame = AudioFrame { - data: std::borrow::Cow::Borrowed(&frame_buf), - sample_rate, - num_channels: channels as u32, - samples_per_channel: samples_per_channel as u32, - }; - if source.capture_frame(&frame).await.is_err() { - break; + let frame = loading_audio_frame(&frame_buf, sample_rate, channels, samples_per_channel); + tokio::select! { + biased; + _ = token.cancelled() => break, + res = source.capture_frame(&frame) => { + if res.is_err() { + break; + } + } } } } @@ -565,7 +615,8 @@ mod tests { let token = CancellationToken::new(); token.cancel(); - let handle = tokio::spawn(run_loading_loop(source, clip, token)); + let loading_loop = Arc::new(Mutex::new(None)); + let handle = tokio::spawn(run_loading_loop(source, clip, token, loading_loop)); let outcome = tokio::time::timeout(std::time::Duration::from_secs(5), handle).await; assert!( diff --git a/src/livekit/client/mod.rs b/src/livekit/client/mod.rs index d5b579e..43e3a57 100644 --- a/src/livekit/client/mod.rs +++ b/src/livekit/client/mod.rs @@ -280,7 +280,11 @@ impl LiveKitClient { #[cfg(test)] pub(crate) async fn has_loading_loop(&self) -> bool { - self.loading_loop.lock().await.is_some() + self.loading_loop + .lock() + .await + .as_ref() + .is_some_and(|existing| !existing.handle.is_finished()) } } diff --git a/src/livekit/client/tests.rs b/src/livekit/client/tests.rs index 0608ded..2782df1 100644 --- a/src/livekit/client/tests.rs +++ b/src/livekit/client/tests.rs @@ -393,10 +393,46 @@ async fn test_start_loading_audio_not_connected_errors() { let config = create_test_config(); let client = LiveKitClient::new(config); - let result = client.start_loading_audio().await; + let err = client + .start_loading_audio() + .await + .expect_err("start_loading_audio should fail when not connected"); + match err { + AppError::BadRequest(msg) => assert!( + msg.contains("active LiveKit connection"), + "unexpected error message: {msg}" + ), + other => panic!("expected BadRequest when not connected, got {other:?}"), + } +} + +#[tokio::test] +async fn test_has_loading_loop_false_when_handle_finished() { + use super::loading_audio::LoadingLoopHandle; + + let config = create_test_config(); + let client = LiveKitClient::new(config); + + let handle = tokio::spawn(async {}); + for _ in 0..32 { + if handle.is_finished() { + break; + } + tokio::task::yield_now().await; + } assert!( - result.is_err(), - "start_loading_audio should fail when not connected" + handle.is_finished(), + "placeholder task should finish promptly" + ); + + *client.loading_loop.lock().await = Some(LoadingLoopHandle { + handle, + token: tokio_util::sync::CancellationToken::new(), + }); + + assert!( + !client.has_loading_loop().await, + "a finished JoinHandle must not count as an active loading loop" ); } @@ -414,7 +450,7 @@ async fn test_start_loading_audio_connected_without_clip_errors() { .expect_err("start_loading_audio should fail without a loading clip"); match err { AppError::BadRequest(msg) => assert!( - msg.contains("no loading audio was configured"), + msg.contains("no loading audio configured"), "unexpected error message: {msg}" ), other => panic!("expected BadRequest for the missing-clip case, got {other:?}"), @@ -563,6 +599,29 @@ async fn livekit_native_start_idempotent_and_disconnect_teardown() { assert!(!client.has_loading_clip()); } +#[tokio::test] +#[ignore = "creates native libwebrtc objects; run isolated via the dedicated CI step (see ci.yml)"] +async fn livekit_native_loading_loop_cleared_after_stop() { + let client = connected_client_with_loading_source().await; + + client + .start_loading_audio() + .await + .expect("start_loading_audio should succeed"); + assert!(client.has_loading_loop().await); + + client.stop_loading_audio().await; + + let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + while tokio::time::Instant::now() < deadline { + if !client.has_loading_loop().await { + return; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + panic!("loading_loop slot must be cleared after stop and fade-out complete"); +} + #[tokio::test] #[ignore = "creates native libwebrtc objects; run isolated via the dedicated CI step (see ci.yml)"] async fn livekit_native_rapid_start_stop_is_clean() { diff --git a/src/livekit/loading_clip.rs b/src/livekit/loading_clip.rs index f645694..1f222ba 100644 --- a/src/livekit/loading_clip.rs +++ b/src/livekit/loading_clip.rs @@ -285,8 +285,10 @@ fn decode_raw_pcm( .ok_or_else(|| "sample_rate is required for raw PCM loading_audio".to_string())?; let channels = channels.unwrap_or(1); - // `channels` defaults to 1 and is validated against {1, 2} by the caller, - // so `frame_bytes` is always non-zero (>= 2) here. + // `channels` is the raw client-supplied value (the caller validates it is + // 1 or 2 only *after* this returns), so `channels: 0` can reach here and + // make `frame_bytes` zero. `is_multiple_of` is panic-safe for a zero + // divisor — unlike `%` — so the alignment check below stays sound. let frame_bytes = 2 * channels as usize; if !bytes.len().is_multiple_of(2) || !bytes.len().is_multiple_of(frame_bytes) { return Err("loading_audio raw PCM length is not frame-aligned".to_string()); @@ -580,6 +582,21 @@ mod tests { assert_eq!(err, "loading_audio raw PCM length is not frame-aligned"); } + #[test] + fn test_zero_channels_raw_pcm_rejected_without_panic() { + // `channels: 0` is the raw client-supplied value; the channel-count + // validation runs only *after* `decode_raw_pcm` returns, so a zero + // reaches the alignment check and makes `frame_bytes` zero. + // `decode_raw_pcm` uses `is_multiple_of` (panic-safe for a zero + // divisor) rather than `%`, so this must reject cleanly, never panic. + let samples: Vec = vec![100; 8000]; + let data = raw_pcm_to_base64(&samples); + + let err = decode_loading_clip(&data, Some("pcm"), Some(16_000), Some(0), None) + .expect_err("raw PCM with channels: 0 must be rejected, not panic"); + assert_eq!(err, "loading_audio raw PCM length is not frame-aligned"); + } + #[test] fn test_sample_rate_too_low_rejected() { // 4 kHz is below LOADING_AUDIO_SAMPLE_RATE_MIN. From 3a764298e0f00ad6a096a3b79cafa71162651856 Mon Sep 17 00:00:00 2001 From: tigranbs Date: Sun, 24 May 2026 23:14:23 -0700 Subject: [PATCH 4/5] refactor(livekit): dedupe AudioSourceOptions into shared helper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five sites constructed the same `AudioSourceOptions { echo_cancellation: false, noise_suppression: false, auto_gain_control: false }` literal — two TTS sites (`setup_audio_publishing`, `process_reconnect`), the loading-audio track publisher, and two test-only injection sites. `auto_gain_control: false` is load-bearing for the loading-audio `volume` feature (AGC would re-normalise loudness and silently undo the configured attenuation), and that rationale was only spelled out at one of the five sites. Introduce `sayna_audio_source_options()` in `client/mod.rs` whose docstring owns the rationale for all three flags. Replace every literal with a call to the helper. The helper is re-exported `pub(crate)` under `#[cfg(test)]` so the cross-module test in `handlers/ws/loading_handler.rs` can reach it without exposing `client/` internals at runtime. Also simplify two `loading_stop` no-op tests in `loading_handler.rs` whose `(message_tx, mut message_rx) + drop(message_tx)` idiom only obscured the intent — the handler takes no sender, so a discarded `_tx` is clearer. The previously-drafted lower-level race tests against `run_loading_loop` were dropped because the existing `livekit_native_rapid_start_stop_is_clean`, `livekit_native_start_idempotent_and_disconnect_teardown`, `livekit_native_loading_loop_cleared_after_stop`, and `livekit_native_drop_cancels_active_loop` already exercise the same races through the real `LiveKitClient` public API. --- src/handlers/ws/loading_handler.rs | 21 ++++---------------- src/livekit/client/audio.rs | 23 +++++++--------------- src/livekit/client/loading_audio.rs | 30 +++++++++++++---------------- src/livekit/client/mod.rs | 22 +++++++++++++++++++++ src/livekit/client/tests.rs | 26 +++++-------------------- src/livekit/mod.rs | 2 ++ 6 files changed, 53 insertions(+), 71 deletions(-) diff --git a/src/handlers/ws/loading_handler.rs b/src/handlers/ws/loading_handler.rs index ce20667..5d1f556 100644 --- a/src/handlers/ws/loading_handler.rs +++ b/src/handlers/ws/loading_handler.rs @@ -312,13 +312,9 @@ mod tests { state_inner.stream_id = Some("test-stream".to_string()); state_inner.livekit_client = Some(Arc::new(RwLock::new(client))); let state = Arc::new(RwLock::new(state_inner)); - let (message_tx, mut message_rx): ( - mpsc::Sender, - mpsc::Receiver, - ) = mpsc::channel(4); + let (_tx, mut message_rx) = mpsc::channel::(4); let continue_processing = handle_loading_stop_message(&state).await; - drop(message_tx); assert!(continue_processing); assert!( @@ -330,9 +326,8 @@ mod tests { #[tokio::test] async fn test_handle_loading_start_message_success_is_silent() { use crate::livekit::loading_clip::make_test_loading_clip; - use crate::livekit::{LiveKitClient, LiveKitConfig}; + use crate::livekit::{LiveKitClient, LiveKitConfig, sayna_audio_source_options}; use livekit::webrtc::audio_source::native::NativeAudioSource; - use livekit::webrtc::prelude::AudioSourceOptions; let clip = make_test_loading_clip(); let mut client = LiveKitClient::new(LiveKitConfig { @@ -350,11 +345,7 @@ mod tests { client.set_loading_audio_clip(clip); let source = Arc::new(NativeAudioSource::new( - AudioSourceOptions { - echo_cancellation: false, - noise_suppression: false, - auto_gain_control: false, - }, + sayna_audio_source_options(), 16_000, 1, 100, // NativeAudioSource queue depth in ms (matches LOADING_AUDIO_QUEUE_SIZE_MS) @@ -389,13 +380,9 @@ mod tests { #[tokio::test] async fn test_handle_loading_stop_message_is_silent_noop() { let state = Arc::new(RwLock::new(ConnectionState::new())); - let (message_tx, mut message_rx): ( - mpsc::Sender, - mpsc::Receiver, - ) = mpsc::channel(4); + let (_tx, mut message_rx) = mpsc::channel::(4); let continue_processing = handle_loading_stop_message(&state).await; - drop(message_tx); assert!(continue_processing); assert!( diff --git a/src/livekit/client/audio.rs b/src/livekit/client/audio.rs index 7f6d9cb..557b5d5 100644 --- a/src/livekit/client/audio.rs +++ b/src/livekit/client/audio.rs @@ -12,11 +12,14 @@ use livekit::options::TrackPublishOptions; use livekit::prelude::{DataPacketKind, Room, RoomEvent}; use livekit::track::{LocalAudioTrack, LocalTrack, TrackSource}; use livekit::webrtc::audio_source::native::NativeAudioSource; -use livekit::webrtc::prelude::{AudioFrame, AudioSourceOptions, RtcAudioSource}; +use livekit::webrtc::prelude::{AudioFrame, RtcAudioSource}; use tokio::sync::{Mutex, mpsc, oneshot}; use tracing::{debug, error, info, warn}; -use super::{LiveKitClient, LiveKitConfig, LiveKitOperation, operation_worker::OperationContext}; +use super::{ + LiveKitClient, LiveKitConfig, LiveKitOperation, operation_worker::OperationContext, + sayna_audio_source_options, +}; use crate::AppError; impl LiveKitClient { @@ -26,16 +29,10 @@ impl LiveKitClient { self.config.sample_rate, self.config.channels ); - let audio_source_options = AudioSourceOptions { - echo_cancellation: false, - noise_suppression: false, - auto_gain_control: false, - }; - let samples_per_frame = (self.config.sample_rate * 10) / 1000; let audio_source = Arc::new(NativeAudioSource::new( - audio_source_options, + sayna_audio_source_options(), self.config.sample_rate, self.config.channels as u32, samples_per_frame, @@ -522,15 +519,9 @@ impl LiveKitClient { } }; - let audio_source_options = AudioSourceOptions { - echo_cancellation: false, - noise_suppression: false, - auto_gain_control: false, - }; - let samples_per_frame = (ctx.config.sample_rate * 10) / 1000; let new_audio_source = Arc::new(NativeAudioSource::new( - audio_source_options, + sayna_audio_source_options(), ctx.config.sample_rate, ctx.config.channels as u32, samples_per_frame, diff --git a/src/livekit/client/loading_audio.rs b/src/livekit/client/loading_audio.rs index c50a43d..b4147d0 100644 --- a/src/livekit/client/loading_audio.rs +++ b/src/livekit/client/loading_audio.rs @@ -19,12 +19,12 @@ use livekit::options::TrackPublishOptions; use livekit::prelude::LocalTrackPublication; use livekit::track::{LocalAudioTrack, LocalTrack, TrackSource}; use livekit::webrtc::audio_source::native::NativeAudioSource; -use livekit::webrtc::prelude::{AudioFrame, AudioSourceOptions, RtcAudioSource}; +use livekit::webrtc::prelude::{AudioFrame, RtcAudioSource}; use tokio::sync::Mutex; use tokio_util::sync::CancellationToken; use tracing::{debug, info, warn}; -use super::LiveKitClient; +use super::{LiveKitClient, sayna_audio_source_options}; use crate::AppError; use crate::livekit::loading_clip::LoadingClip; @@ -155,17 +155,8 @@ pub(super) async fn publish_loading_audio_track( ), AppError, > { - // `auto_gain_control: false` is load-bearing for the `volume` feature: - // WebRTC AGC would re-normalize loudness and silently undo the configured - // volume attenuation already applied to the clip's samples. - let audio_source_options = AudioSourceOptions { - echo_cancellation: false, - noise_suppression: false, - auto_gain_control: false, - }; - let source = Arc::new(NativeAudioSource::new( - audio_source_options, + sayna_audio_source_options(), sample_rate, channels as u32, LOADING_AUDIO_QUEUE_SIZE_MS, @@ -600,11 +591,7 @@ mod tests { let clip = Arc::new(crate::livekit::loading_clip::make_test_loading_clip()); let source = Arc::new(NativeAudioSource::new( - AudioSourceOptions { - echo_cancellation: false, - noise_suppression: false, - auto_gain_control: false, - }, + sayna_audio_source_options(), clip.sample_rate(), clip.channels() as u32, LOADING_AUDIO_QUEUE_SIZE_MS, @@ -628,4 +615,13 @@ mod tests { "run_loading_loop task must not panic" ); } + + // NOTE: integration-level race/stress coverage of `start_loading_audio` / + // `stop_loading_audio` / `disconnect` lives in `client/tests.rs` + // (`livekit_native_rapid_start_stop_is_clean`, + // `livekit_native_start_idempotent_and_disconnect_teardown`, + // `livekit_native_loading_loop_cleared_after_stop`, + // `livekit_native_drop_cancels_active_loop`). Those use the real public + // API; replicating them here against the lower-level `run_loading_loop` + // would only duplicate behavioural assertions. } diff --git a/src/livekit/client/mod.rs b/src/livekit/client/mod.rs index 43e3a57..84ef2ca 100644 --- a/src/livekit/client/mod.rs +++ b/src/livekit/client/mod.rs @@ -23,6 +23,7 @@ use std::sync::atomic::AtomicBool; use livekit::prelude::{LocalTrackPublication, Room, RoomEvent}; use livekit::track::LocalAudioTrack; use livekit::webrtc::audio_source::native::NativeAudioSource; +use livekit::webrtc::prelude::AudioSourceOptions; use tokio::sync::{Mutex, mpsc}; use tracing::warn; @@ -107,6 +108,27 @@ pub(crate) const RELIABLE_BUFFER_THRESHOLD_BYTES: u64 = 200 * 1024 * 1024; /// Maximum audio queue size based on duration (2 seconds worth of frames at 100 frames/sec) pub(crate) const MAX_AUDIO_QUEUE_SIZE: usize = 200; +/// Canonical `AudioSourceOptions` for every audio source Sayna publishes. +/// +/// Every flag is `false` because Sayna feeds already-mastered PCM into LiveKit +/// — the server's job is to deliver bytes faithfully, not to re-process them: +/// * `echo_cancellation` is a capture-side feature for microphone input and +/// would only distort a synthesized output stream. +/// * `noise_suppression` would re-filter audio that the TTS engine (or the +/// client-supplied loading clip) already produced clean. +/// * `auto_gain_control` is **load-bearing** for the loading-audio track: the +/// loading clip's loudness is set by scaling its samples at decode time +/// (`volume` config field), and AGC would re-normalise that loudness and +/// silently undo the attenuation. It is equally undesirable on the TTS +/// track, whose levels are set by the TTS provider. +pub(crate) fn sayna_audio_source_options() -> AudioSourceOptions { + AudioSourceOptions { + echo_cancellation: false, + noise_suppression: false, + auto_gain_control: false, + } +} + /// LiveKit client for handling audio streaming and publishing. /// /// Optimized for low latency with queue-based operations to eliminate lock contention. diff --git a/src/livekit/client/tests.rs b/src/livekit/client/tests.rs index 2782df1..6a804dc 100644 --- a/src/livekit/client/tests.rs +++ b/src/livekit/client/tests.rs @@ -509,27 +509,16 @@ async fn test_disconnect_clears_loading_fields() { async fn livekit_native_two_independent_audio_sources() { use livekit::track::LocalAudioTrack; use livekit::webrtc::audio_source::native::NativeAudioSource; - use livekit::webrtc::prelude::{AudioSourceOptions, RtcAudioSource}; - - // `AudioSourceOptions` is not `Clone`; construct one per source. - let tts_options = AudioSourceOptions { - echo_cancellation: false, - noise_suppression: false, - auto_gain_control: false, - }; - let loading_options = AudioSourceOptions { - echo_cancellation: false, - noise_suppression: false, - auto_gain_control: false, - }; + use livekit::webrtc::prelude::RtcAudioSource; // TTS-style source at 24 kHz mono. - let tts_source = NativeAudioSource::new(tts_options, 24_000, 1, 240); + let tts_source = NativeAudioSource::new(super::sayna_audio_source_options(), 24_000, 1, 240); let tts_track = LocalAudioTrack::create_audio_track("tts-audio", RtcAudioSource::Native(tts_source)); // Loading-style source at a different sample rate, 16 kHz mono. - let loading_source = NativeAudioSource::new(loading_options, 16_000, 1, 160); + let loading_source = + NativeAudioSource::new(super::sayna_audio_source_options(), 16_000, 1, 160); let loading_track = LocalAudioTrack::create_audio_track( "loading-audio", RtcAudioSource::Native(loading_source), @@ -547,7 +536,6 @@ async fn livekit_native_two_independent_audio_sources() { /// which unit tests do not have. async fn connected_client_with_loading_source() -> LiveKitClient { use livekit::webrtc::audio_source::native::NativeAudioSource; - use livekit::webrtc::prelude::AudioSourceOptions; let clip = crate::livekit::loading_clip::make_test_loading_clip(); @@ -556,11 +544,7 @@ async fn connected_client_with_loading_source() -> LiveKitClient { client.set_loading_audio_clip(clip); let source = Arc::new(NativeAudioSource::new( - AudioSourceOptions { - echo_cancellation: false, - noise_suppression: false, - auto_gain_control: false, - }, + super::sayna_audio_source_options(), 16_000, 1, super::loading_audio::LOADING_AUDIO_QUEUE_SIZE_MS, diff --git a/src/livekit/mod.rs b/src/livekit/mod.rs index 3eb7425..4fef450 100644 --- a/src/livekit/mod.rs +++ b/src/livekit/mod.rs @@ -78,6 +78,8 @@ pub mod sip_handler; mod types; // Re-export public types and traits +#[cfg(test)] +pub(crate) use client::sayna_audio_source_options; pub use client::{AudioCallback, DataCallback, DataMessage, LiveKitClient}; pub use loading_clip::{LoadingClip, decode_loading_clip}; pub use manager::LiveKitManager; From c593741abdfcd0d3779c96b0c3c4d1aaf2064697 Mon Sep 17 00:00:00 2001 From: tigranbs Date: Sun, 24 May 2026 23:46:27 -0700 Subject: [PATCH 5/5] test(loading): quarantine handler test that creates real NativeAudioSource test_handle_loading_start_message_success_is_silent constructed a real libwebrtc NativeAudioSource and spawned the loading-audio loop, but ran in the default multi-threaded cargo test pass. The project quarantines such tests behind a livekit_native_ prefix and an #[ignore] attribute because libwebrtc's lazily-initialised global runtime intermittently segfaults under thread concurrency (see src/livekit/client/tests.rs:499-506 and .github/workflows/ci.yml). This test was an outlier. Rename it with the livekit_native_ prefix and add the same #[ignore] attribute as its peers, so the dedicated single-threaded CI step picks it up alongside the existing six native tests. --- src/handlers/ws/loading_handler.rs | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/handlers/ws/loading_handler.rs b/src/handlers/ws/loading_handler.rs index 5d1f556..8dc5959 100644 --- a/src/handlers/ws/loading_handler.rs +++ b/src/handlers/ws/loading_handler.rs @@ -323,8 +323,14 @@ mod tests { ); } + // This test constructs a real libwebrtc `NativeAudioSource` and so wears + // the `livekit_native_` quarantine prefix described in + // `src/livekit/client/tests.rs:499-506`: it is `#[ignore]`d out of the + // default `cargo test` run and executed isolated by the dedicated CI step + // in `.github/workflows/ci.yml`. #[tokio::test] - async fn test_handle_loading_start_message_success_is_silent() { + #[ignore = "creates native libwebrtc objects; run isolated via the dedicated CI step (see ci.yml)"] + async fn livekit_native_handle_loading_start_message_success_is_silent() { use crate::livekit::loading_clip::make_test_loading_clip; use crate::livekit::{LiveKitClient, LiveKitConfig, sayna_audio_source_options}; use livekit::webrtc::audio_source::native::NativeAudioSource;