Open Jarvis is a voice-first AI assistant for your Windows laptop. Say "hey Jarvis", ask about whatever is on your screen, and hear the answer. It reads the accessibility tree, knows where keyboard focus is, and can carry out multi-step tasks for you when you ask.
It speaks with a local neural voice and listens for its wake word entirely on your machine, so nothing is streamed to a cloud service until you actually ask a question.
Open Jarvis started as an accessibility tool and still works beside a screen reader: NVDA users can keep their own voice and rate by setting voice_backend to nvda.
Read PLAN.md for the product and engineering plan and CONTRACTS.md for module interfaces.
Say "hey Jarvis", wait for the listen tone, then ask. There is no key to hold: Open Jarvis works out when you have finished by listening for a pause of about a second, then sends the question. If you say the phrase and then say nothing, it stands down after three seconds.
Detection runs on this laptop through openWakeWord's pretrained model. The audio never leaves the machine and nothing is sent anywhere until the phrase actually fires.
The microphone stays open while Open Jarvis is running, which is a bigger promise than push-to-talk makes. Two things bound it: the wake word listener closes the microphone entirely at privacy level three, and push-to-talk still works, so you can set wake_word_enabled to false and go back to holding a key.
Tune wake_word_threshold if needed: lower it if the phrase is missed, raise it if Open Jarvis wakes up on its own.
Every shortcut uses Ctrl and Alt together so it never collides with NVDA or JAWS commands, which use Insert or Caps Lock.
| Keys | What happens |
|---|---|
| Ctrl+Alt+Space, hold | Push to talk. A short tone plays when the microphone opens. Speak, then release to send. Space is not typed into the app you are in. |
| Ctrl+Alt+C | Screen curtain on or off. The display goes black for anyone watching. Open Jarvis keeps working. |
| Ctrl+Alt+P | Cycle the privacy level: full, text only, offline. Open Jarvis announces the new level. |
| Ctrl+Alt+R | Repeat the last answer. |
| Ctrl+Alt+S | Stop speaking, and cancel a task that is running. |
| Ctrl+Alt+Y | Yes, when Open Jarvis asks before an irreversible action. |
| Ctrl+Alt+N | No, when Open Jarvis asks before an irreversible action. |
When Open Jarvis asks "say yes or no", the microphone opens by itself for about four seconds after the question, with the listen tone. Just say yes or no, no key needed. Ctrl+Alt+Y or Ctrl+Alt+N also work and answer instantly. Only a short, clear yes counts as yes; silence or anything else is treated as no.
All shortcuts can be changed in app/data/settings.json (see Settings below).
Open Jarvis uses short tones instead of a visual spinner so you always know what it is doing.
| Tone | Meaning |
|---|---|
| Listen | Microphone is open. Speak now. |
| Think | Your words were sent. Waiting for the answer. |
| Done | The answer or task finished. |
| Error | Something failed. Open Jarvis says what. |
| Curtain on / off | Screen curtain toggled. |
| Privacy | Privacy level changed. |
| Offline | No network. Only local narration is available. |
Questions about the screen
- "Where am I?" App, window, focused control, and what is around it.
- "What just happened?" What changed since you last asked: a dialog opened, focus moved, a page loaded.
- "How do I attach a file here?" The next step or two with exact keys, then it waits for you to do it.
- "Why did that not work?" Most likely reason and the fix.
- "Read this." The text on screen in reading order.
Tasks it does for you
Ask with words like "do it", "for me", "you click it". Open Jarvis then works in rounds: act, wait for the page to settle, look at the screen again, continue. It says one short line per round.
- "Open YouTube and search and play a new Hindi song."
- "Open Flipkart and search for a boAt watch under two thousand rupees."
- "Click the Send button for me."
- "Type my address into this box." (It will never type into password, OTP, PIN, or card fields.)
- "Read the first three names on my desktop, then open Notepad and type them." Open Jarvis remembers what it read and carries it into the next app.
Your own data is yours. Open Jarvis will read your contacts, chats, prices, or files aloud and copy them where you ask, the same as a sighted helper would. It only refuses passwords, OTPs, PINs, and card numbers.
Before anything irreversible, such as send, submit, post, pay, delete, or sign out, Open Jarvis asks, then listens for a spoken yes or no on its own. Ctrl+Alt+Y and Ctrl+Alt+N work too. It stops after eight rounds, or as soon as it notices it is repeating the same step, and hands over to you when only you can continue, for example at a login screen, an OTP, or a CAPTCHA.
Offline
At privacy level three, or when the network is down, Open Jarvis still answers "where am I" from the accessibility tree alone.
| Level | What leaves the laptop | Use it when |
|---|---|---|
| 1 Full | Screen text as a numbered list plus a downscaled picture of the current window | Learning, apps with unlabeled buttons, anything visual |
| 2 Text only | Screen text only, no picture | Offices, banking, documents open on screen |
| 3 Offline | Nothing | No network, or maximum privacy |
Password, OTP, PIN, and card fields are masked at every level. The screen curtain is independent of these levels and is meant to stay on whenever other people are around. Screenshots still work under the curtain because Open Jarvis renders the window directly rather than grabbing the screen.
Open Jarvis is two processes: a Worker that holds the model key, and the client that listens and speaks. Both must be running. Every command below assumes the repo is at C:\OpenJarvis — substitute your own path.
Requirements: Windows 11, Python 3.12, Node 20 or newer. A microphone. Speakers or headphones.
1. Install Python 3.12 from python.org, ticking "Add python.exe to PATH" in the installer. Then open a new terminal and check:
python --version # must print 3.12.xIf it prints nothing, opens the Microsoft Store, or says "Python was not found", PATH is not set. The Store stub at %LOCALAPPDATA%\Microsoft\WindowsApps\python.exe is not a real Python.
2. Allow desktop apps to use your microphone. Open Jarvis cannot hear anything until this is on, and the failure looks like a broken audio driver rather than a permissions problem.
Settings → Privacy & security → Microphone → scroll past the list of Store apps → turn on "Let desktop apps access your microphone".
There are three separate switches on that page. The top two can both read "Allow" while this third one is off. If it is off, every audio backend fails with PaErrorCode -9999.
3. Install the Python packages:
cd C:\OpenJarvis
pip install -r requirements.txt4. Download the voice models (about 350 MB, once):
python -m app.speech.fetch_modelsThis puts the Kokoro weights in app/data/voices/ and the wake word model in openWakeWord's cache. Everything runs offline afterwards. Skip this step and Open Jarvis still starts, but with the robotic Windows voice and push-to-talk only.
5. Add your Gemini API key. Copy .env.example to .env and fill in GEMINI_API_KEY. Then give the Worker its own copy, which is where wrangler reads secrets from:
cd C:\OpenJarvis\worker
npm install
node -e "const fs=require('fs');const l=fs.readFileSync('../.env','utf8').split(/\r?\n/).find(x=>x.startsWith('GEMINI_API_KEY='));fs.writeFileSync('.dev.vars',l+'\n')"Two terminals, both stay open.
Terminal 1 — the Worker. Note the cd into worker: running npm run dev from the repo root fails with Missing script: "dev", because the only package.json is in that subfolder.
cd C:\OpenJarvis\worker
npm run devWait for Ready on http://127.0.0.1:8787.
Terminal 2 — the client:
cd C:\OpenJarvis
python -m app.mainIt says "At your service." The first start takes a few seconds while the voice model warms up. If anything is wrong — a hotkey clash, no Worker, not running as administrator — it says that too, straight after.
To stop either one, press Ctrl+C in its terminal.
app\data\openjarvis.log should contain all three of these, and no microphone error after the wake word line:
Hotkeys active: push-to-talk ctrl+alt+space
wake word listening for 'hey_jarvis' at threshold 0.50
speech backend=kokoro wake=on worker=http://127.0.0.1:8787 healthy=True
| What you see instead | What it means |
|---|---|
healthy=False |
Terminal 1 is not running, or died. Start the Worker. |
backend=sapi |
Kokoro did not load. Run python -m app.speech.fetch_models. |
could not open the microphone |
The Windows toggle in step 2 is off. |
wake=off |
openwakeword is missing. Push-to-talk still works. |
Then say "hey Jarvis", wait for the tone, and ask "where am I?".
After pulling new code, stop the client with Ctrl+C and start it again. The Worker reloads on its own.
By default Open Jarvis speaks with Kokoro, an 82M-parameter neural voice that runs locally on the CPU. The default voice is bm_george, a British male voice chosen for the calm, clipped register people recognise as an AI assistant. Other British male options are bm_daniel, bm_lewis, and bm_fable; the full list is in Kokoro's VOICES.md. Kokoro is Apache 2.0, so there is nothing to license.
To be clear about what this is: it is a British AI-assistant voice, not a clone of the actor who played JARVIS. Cloning a real person's voice from film audio is what right-of-publicity statutes like Tennessee's ELVIS Act exist to stop, and it is not something you want in a public demo.
Set voice_backend to choose:
| Value | What speaks |
|---|---|
kokoro |
The local neural voice. The default. |
auto |
NVDA if it is running, otherwise the Windows voices. |
nvda |
NVDA only, using your own voice and rate. |
sapi |
The Windows voices. |
Every choice falls back rather than going silent: if the Kokoro model files are missing, Open Jarvis logs it and uses the Windows voice.
For Indian English and Hindi through SAPI, add English (India) and Hindi with speech in Windows Settings under Time and Language, then Language and region. The voices are Ravi and Heera for English, Hemant and Kalpana for Hindi.
Settings live in app/data/settings.json, created on first run. Every key is optional.
| Key | Default | Meaning |
|---|---|---|
worker_url |
http://127.0.0.1:8787 |
Where the Worker runs. Also settable with the WORKER_URL environment variable. |
privacy_level |
1 |
1 full, 2 text only, 3 offline |
language |
en-IN |
Answer language hint. Open Jarvis answers in the language you speak. |
speech_rate |
0 |
Windows voice speed, -10 slow to 10 fast. Ignored when speaking through NVDA. |
ptt_hotkey |
ctrl+alt+space |
Push to talk |
curtain_hotkey |
ctrl+alt+c |
Screen curtain |
privacy_hotkey |
ctrl+alt+p |
Privacy level |
repeat_hotkey |
ctrl+alt+r |
Repeat |
stop_hotkey |
ctrl+alt+s |
Stop |
yes_hotkey |
ctrl+alt+y |
Confirm |
no_hotkey |
ctrl+alt+n |
Decline |
max_image_width |
1024 |
Screenshot width in pixels before sending |
voice_backend |
kokoro |
kokoro, auto, nvda, or sapi. See Voices above. |
kokoro_voice |
bm_george |
Which Kokoro voice speaks |
kokoro_speed |
1.0 |
Speaking rate, 0.5 to 2.0 |
kokoro_model_path |
blank | Override the path to kokoro-v1.0.onnx |
kokoro_voices_path |
blank | Override the path to voices-v1.0.bin |
wake_word_enabled |
true |
Listen for "hey Jarvis". Set false for push-to-talk only. |
wake_word |
hey_jarvis |
Any openWakeWord model name |
wake_word_threshold |
0.5 |
0.1 to 0.99. Lower catches more, raises false wakes. |
Hotkeys are written as lowercase key names joined with plus. Do not use Insert or Caps Lock; Open Jarvis warns at startup if a hotkey clashes with a screen reader.
Everything is under app/data/, which is not committed:
settings.jsonyour settingsopenjarvis.sqliterecent turns, remembered labels for unlabeled controlsopenjarvis.logwhat happened, including every action and its result. Screen content is never logged.device_ida random id so the Worker can apply a daily limit per install
app/ Windows client (Python 3.12)
capture/ UI Automation tree, screenshot, screen curtain, privacy filter, OCR
speech/ microphone, speech out (NVDA or Windows voices), earcons
brain/ Worker client, action runner, orchestrator, system prompt
store/ local SQLite for settings, turns, remembered labels
worker/ Cloudflare Worker that holds the model keys and talks to the model
sandbox/ safe practice pages
tests/ pytest, no hardware needed by default
bench/ latency measurements and end-to-end checks
pytest # pure logic, fast, no hardware
pytest -m hardware # needs a display, mic, speakers
python bench/latency.py # capture, screenshot, curtain timings against the budgets
python bench/e2e_voice.py # synthesised speech through capture, Worker, and actions on Notepad
python bench/e2e_task.py "open youtube and search and play a new hindi song"
Worker tests: cd worker && npm test.
- "I do not know that action." The client is older than the Worker. Restart
python -m app.main. - "I cannot reach the tutor service." The Worker is not running or
worker_urlis wrong. Start it withnpm run devinworker/. - "I did not catch that." The recording was under a third of a second or silent. Hold the key a little longer and speak.
- Hotkeys work inside apps but not on the desktop or home screen. Some manufacturer tools, for example MSI Dragon Center Voice Control, keep an invisible administrator-level window alive. When you minimise everything, Windows puts that window in front, and Windows does not let a normal program's keyboard hook see keystrokes headed to an administrator window. Fix: start Open Jarvis with
run_admin.cmd(runs it as administrator), or stop that tool's service. Open Jarvis says a short note at startup when it is not running as administrator. - Nothing happens on the hotkey anywhere. Another program running as administrator has the focus. Use
run_admin.cmd, or change the hotkey in settings. - "I could not open the microphone", or the log says
wake word could not open the microphone. Windows has two separate microphone switches and the second one is easy to miss. Open Settings, Privacy and security, Microphone, and turn on "Let desktop apps access your microphone". The toggle above it can read "Allow" while this one is off, and desktop apps like Open Jarvis are blocked by that one. Every audio backend fails withPaErrorCode -9999until it is on. - Open Jarvis does not react to "hey Jarvis". Check the log for
wake word listening. If it says the wake word is off, runpython -m app.speech.fetch_models. If it is listening but never fires, lowerwake_word_thresholdto about0.35. The phrase on its own, "Jarvis", also works but is missed more often. - Open Jarvis wakes up on its own. Raise
wake_word_thresholdtoward0.7. - It cuts me off mid-question. It ends the turn after about a second of quiet. Speak without long pauses, or set
wake_word_enabledtofalseand hold the key instead. - The voice is the old robotic Windows one. The Kokoro model files are missing or failed to load; the log line starting
Kokoro voice unavailablesays why. Runpython -m app.speech.fetch_models. - Slow answers. Chrome with many tabs produces a large tree. Close tabs you do not need, or use privacy level two so no picture is sent.