Skip to content

Add xAI realtime provider tools#1583

Open
rosetta-livekit-bot[bot] wants to merge 1 commit into
brian/provider-tool-basefrom
beveling-sounder-finesse
Open

Add xAI realtime provider tools#1583
rosetta-livekit-bot[bot] wants to merge 1 commit into
brian/provider-tool-basefrom
beveling-sounder-finesse

Conversation

@rosetta-livekit-bot
Copy link
Copy Markdown
Contributor

@rosetta-livekit-bot rosetta-livekit-bot Bot commented May 22, 2026

Summary

  • Add xAI ProviderTool classes for Realtime API web search, X search, and file search
  • Serialize xAI provider tools into realtime session.update payloads
  • Keep OpenAI Realtime function-tool retention robust when provider tools are present

Stacked on #1576 for the ProviderTool base abstraction, mirroring #1582's provider-tool pattern for xAI.

Testing

  • pnpm build
  • pnpm exec prettier --check .changeset/xai-provider-tools.md plugins/xai/src/index.ts plugins/xai/src/realtime/realtime_model.ts plugins/xai/src/tools.ts plugins/openai/src/realtime/realtime_model.ts

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented May 22, 2026

🦋 Changeset detected

Latest commit: bd921b1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 31 packages
Name Type
@livekit/agents Major
@livekit/agents-plugin-google Major
@livekit/agents-plugin-openai Major
@livekit/agents-plugin-xai Major
@livekit/agents-plugin-anam Major
@livekit/agents-plugin-assemblyai Major
@livekit/agents-plugin-baseten Major
@livekit/agents-plugin-bey Major
@livekit/agents-plugin-cartesia Major
@livekit/agents-plugin-cerebras Major
@livekit/agents-plugin-deepgram Major
@livekit/agents-plugin-elevenlabs Major
@livekit/agents-plugin-fishaudio Major
@livekit/agents-plugin-hedra Major
@livekit/agents-plugin-hume Major
@livekit/agents-plugin-inworld Major
@livekit/agents-plugin-lemonslice Major
@livekit/agents-plugin-liveavatar Major
@livekit/agents-plugin-livekit Major
@livekit/agents-plugin-minimax Major
@livekit/agents-plugin-mistral Major
@livekit/agents-plugin-mistralai Major
@livekit/agents-plugin-neuphonic Major
@livekit/agents-plugin-phonic Major
@livekit/agents-plugin-resemble Major
@livekit/agents-plugin-rime Major
@livekit/agents-plugin-runway Major
@livekit/agents-plugin-sarvam Major
@livekit/agents-plugin-silero Major
@livekit/agents-plugins-test Major
@livekit/agents-plugin-trugen Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

@rosetta-livekit-bot rosetta-livekit-bot Bot force-pushed the beveling-sounder-finesse branch from 42b9982 to bd921b1 Compare May 22, 2026 19:08
@rosetta-livekit-bot rosetta-livekit-bot Bot changed the title feat(xai): add realtime provider tools Add xAI realtime provider tools May 22, 2026
@rosetta-livekit-bot rosetta-livekit-bot Bot changed the base branch from main to brian/provider-tool-base May 22, 2026 19:08
@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ toubatbrian
❌ claude
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

View 5 additional findings in Devin Review.

Open in Devin Review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Reconnect skips restoring xAI provider tools when no function tools are present

The reconnect() method in OpenAIRealtimeSession at plugins/openai/src/realtime/realtime_model.ts:1002 only checks Object.keys(this._tools.functionTools).length > 0 before sending the tools update event. Since functionTools only returns function tools (not provider tools), if an xAI session is configured with only provider tools (e.g., WebSearch, XSearch, FileSearch) and no function tools, those tools will not be sent to the server during reconnection. After the default 20-minute maxSessionDuration triggers a reconnect, the xAI provider tools are silently lost.

Reconnect condition only checks function tools

At plugins/openai/src/realtime/realtime_model.ts:1002-1004:

if (Object.keys(this._tools.functionTools).length > 0) {
  events.push(this.createToolsUpdateEvent(this._tools));
}

functionTools (agents/src/llm/tool_context.ts:287-289) only returns entries from _functionToolsMap, not _providerTools. The xAI RealtimeSession overrides createToolsUpdateEvent to append provider tools, but that method is never called when this condition is false.

(Refers to lines 1002-1004)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant