Agentic AI for XR — an open-source foundation for multimodal, real-time conversational AI in the NVIDIA CloudXR ecosystem.
XR AI connects web, Android, iOS/visionOS, and native clients to a shared media hub, GPU-accelerated AI services, tool-using agents, and optional CloudXR remote rendering. Agents can see and hear what a participant experiences, call native tools, and return audio or data to the same participant.
This project is a public beta. APIs and behavior may change as it evolves.
Refer to the versioned documentation for setup, requirements, credentials, networking, architecture, and troubleshooting. The landing page opens the newest complete documentation set; a release takes precedence once it contains the current entry points. Start with:
- Set up with a coding agent
- Build your own application
- Manual quickstart
- System requirements
- Architecture
The site also publishes the current main branch and release-tagged versions.
Build an application in the XR AI source tree so its unpublished SDK
dependencies resolve from their repository paths. Start from
agent-samples/simple-vlm-example/, copy it to apps/<your-app>/, keep only
the SDK layers and services the application needs, and follow the
build-your-application guide
through scaffolding, startup, and hardware-free verification.
| Sample | Purpose |
|---|---|
model-servers |
Start and persist the shared model stack |
model-servers-nim |
Start the shared model stack using NVIDIA NIM |
simple-vlm-example |
Voice and text questions about the current camera frame |
lab-instrument-monitoring |
Marker-associated visual monitoring with a foreground voice agent |
tea-making-sample |
Guided workflow with visual evidence and background observations |
xr-render-demo |
Voice-driven CloudXR scene manipulation |
Each sample README gives the shortest runnable command sequence. The linked documentation contains architecture, behavior, configuration, output contracts, evaluation, and adaptation guidance.
| Directory | Contents |
|---|---|
client-samples/ |
Platform clients and shared StreamKit implementations |
agent-sdk/ |
Hub IPC, model clients, runtime, tools, voice, and web events |
agent-samples/ |
Runnable agent stacks |
model-server-samples/ |
Shared model-server launch samples |
apps/ |
Application workspaces outside repository sample tooling |
services/ |
Hub, model servers, and typed capability services |
utils/ |
Launcher, logging, VAD, vLLM, and voice-gate utilities |
tests/ |
Cross-package and integration tests |
docs/source/ |
Canonical user and contributor documentation |
skills/ |
Setup skills for coding agents |
For repository constraints and dependency boundaries, refer to
AGENTS.md and DEPENDENCIES.md.
Refer to CONTRIBUTING.md for the contribution process and
tests/README.md for the shortest test commands. Refer to
SECURITY.md to report security issues.
XR AI is licensed under Apache-2.0.
Third-party license and attribution texts are preserved under
third_party_licenses/. These files retain their
upstream terms; refer to License texts
for details.
Refer to THIRD_PARTY_NOTICES.md for each component's
applicable license and third_party_licenses/
for the preserved texts.