Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Open Jarvis

Open Jarvis is a voice-first AI assistant for your Windows laptop. Say "hey Jarvis", ask about whatever is on your screen, and hear the answer. It reads the accessibility tree, knows where keyboard focus is, and can carry out multi-step tasks for you when you ask.

It speaks with a local neural voice and listens for its wake word entirely on your machine, so nothing is streamed to a cloud service until you actually ask a question.

Open Jarvis started as an accessibility tool and still works beside a screen reader: NVDA users can keep their own voice and rate by setting voice_backend to nvda.

Read PLAN.md for the product and engineering plan and CONTRACTS.md for module interfaces.

Saying "hey Jarvis"

Say "hey Jarvis", wait for the listen tone, then ask. There is no key to hold: Open Jarvis works out when you have finished by listening for a pause of about a second, then sends the question. If you say the phrase and then say nothing, it stands down after three seconds.

Detection runs on this laptop through openWakeWord's pretrained model. The audio never leaves the machine and nothing is sent anywhere until the phrase actually fires.

The microphone stays open while Open Jarvis is running, which is a bigger promise than push-to-talk makes. Two things bound it: the wake word listener closes the microphone entirely at privacy level three, and push-to-talk still works, so you can set wake_word_enabled to false and go back to holding a key.

Tune wake_word_threshold if needed: lower it if the phrase is missed, raise it if Open Jarvis wakes up on its own.

Keyboard shortcuts

Every shortcut uses Ctrl and Alt together so it never collides with NVDA or JAWS commands, which use Insert or Caps Lock.

Keys What happens
Ctrl+Alt+Space, hold Push to talk. A short tone plays when the microphone opens. Speak, then release to send. Space is not typed into the app you are in.
Ctrl+Alt+C Screen curtain on or off. The display goes black for anyone watching. Open Jarvis keeps working.
Ctrl+Alt+P Cycle the privacy level: full, text only, offline. Open Jarvis announces the new level.
Ctrl+Alt+R Repeat the last answer.
Ctrl+Alt+S Stop speaking, and cancel a task that is running.
Ctrl+Alt+Y Yes, when Open Jarvis asks before an irreversible action.
Ctrl+Alt+N No, when Open Jarvis asks before an irreversible action.

When Open Jarvis asks "say yes or no", the microphone opens by itself for about four seconds after the question, with the listen tone. Just say yes or no, no key needed. Ctrl+Alt+Y or Ctrl+Alt+N also work and answer instantly. Only a short, clear yes counts as yes; silence or anything else is treated as no.

All shortcuts can be changed in app/data/settings.json (see Settings below).

Sounds

Open Jarvis uses short tones instead of a visual spinner so you always know what it is doing.

Tone Meaning
Listen Microphone is open. Speak now.
Think Your words were sent. Waiting for the answer.
Done The answer or task finished.
Error Something failed. Open Jarvis says what.
Curtain on / off Screen curtain toggled.
Privacy Privacy level changed.
Offline No network. Only local narration is available.

What you can ask

Questions about the screen

  • "Where am I?" App, window, focused control, and what is around it.
  • "What just happened?" What changed since you last asked: a dialog opened, focus moved, a page loaded.
  • "How do I attach a file here?" The next step or two with exact keys, then it waits for you to do it.
  • "Why did that not work?" Most likely reason and the fix.
  • "Read this." The text on screen in reading order.

Tasks it does for you

Ask with words like "do it", "for me", "you click it". Open Jarvis then works in rounds: act, wait for the page to settle, look at the screen again, continue. It says one short line per round.

  • "Open YouTube and search and play a new Hindi song."
  • "Open Flipkart and search for a boAt watch under two thousand rupees."
  • "Click the Send button for me."
  • "Type my address into this box." (It will never type into password, OTP, PIN, or card fields.)
  • "Read the first three names on my desktop, then open Notepad and type them." Open Jarvis remembers what it read and carries it into the next app.

Your own data is yours. Open Jarvis will read your contacts, chats, prices, or files aloud and copy them where you ask, the same as a sighted helper would. It only refuses passwords, OTPs, PINs, and card numbers.

Before anything irreversible, such as send, submit, post, pay, delete, or sign out, Open Jarvis asks, then listens for a spoken yes or no on its own. Ctrl+Alt+Y and Ctrl+Alt+N work too. It stops after eight rounds, or as soon as it notices it is repeating the same step, and hands over to you when only you can continue, for example at a login screen, an OTP, or a CAPTCHA.

Offline

At privacy level three, or when the network is down, Open Jarvis still answers "where am I" from the accessibility tree alone.

Privacy levels

Level What leaves the laptop Use it when
1 Full Screen text as a numbered list plus a downscaled picture of the current window Learning, apps with unlabeled buttons, anything visual
2 Text only Screen text only, no picture Offices, banking, documents open on screen
3 Offline Nothing No network, or maximum privacy

Password, OTP, PIN, and card fields are masked at every level. The screen curtain is independent of these levels and is meant to stay on whenever other people are around. Screenshots still work under the curtain because Open Jarvis renders the window directly rather than grabbing the screen.

Turning it on

Open Jarvis is two processes: a Worker that holds the model key, and the client that listens and speaks. Both must be running. Every command below assumes the repo is at C:\OpenJarvis — substitute your own path.

First time only

Requirements: Windows 11, Python 3.12, Node 20 or newer. A microphone. Speakers or headphones.

1. Install Python 3.12 from python.org, ticking "Add python.exe to PATH" in the installer. Then open a new terminal and check:

python --version        # must print 3.12.x

If it prints nothing, opens the Microsoft Store, or says "Python was not found", PATH is not set. The Store stub at %LOCALAPPDATA%\Microsoft\WindowsApps\python.exe is not a real Python.

2. Allow desktop apps to use your microphone. Open Jarvis cannot hear anything until this is on, and the failure looks like a broken audio driver rather than a permissions problem.

Settings → Privacy & security → Microphone → scroll past the list of Store apps → turn on "Let desktop apps access your microphone".

There are three separate switches on that page. The top two can both read "Allow" while this third one is off. If it is off, every audio backend fails with PaErrorCode -9999.

3. Install the Python packages:

cd C:\OpenJarvis
pip install -r requirements.txt

4. Download the voice models (about 350 MB, once):

python -m app.speech.fetch_models

This puts the Kokoro weights in app/data/voices/ and the wake word model in openWakeWord's cache. Everything runs offline afterwards. Skip this step and Open Jarvis still starts, but with the robotic Windows voice and push-to-talk only.

5. Add your Gemini API key. Copy .env.example to .env and fill in GEMINI_API_KEY. Then give the Worker its own copy, which is where wrangler reads secrets from:

cd C:\OpenJarvis\worker
npm install
node -e "const fs=require('fs');const l=fs.readFileSync('../.env','utf8').split(/\r?\n/).find(x=>x.startsWith('GEMINI_API_KEY='));fs.writeFileSync('.dev.vars',l+'\n')"

Every time you start it

Two terminals, both stay open.

Terminal 1 — the Worker. Note the cd into worker: running npm run dev from the repo root fails with Missing script: "dev", because the only package.json is in that subfolder.

cd C:\OpenJarvis\worker
npm run dev

Wait for Ready on http://127.0.0.1:8787.

Terminal 2 — the client:

cd C:\OpenJarvis
python -m app.main

It says "At your service." The first start takes a few seconds while the voice model warms up. If anything is wrong — a hotkey clash, no Worker, not running as administrator — it says that too, straight after.

To stop either one, press Ctrl+C in its terminal.

Check it actually came up

app\data\openjarvis.log should contain all three of these, and no microphone error after the wake word line:

Hotkeys active: push-to-talk ctrl+alt+space
wake word listening for 'hey_jarvis' at threshold 0.50
speech backend=kokoro wake=on worker=http://127.0.0.1:8787 healthy=True
What you see instead What it means
healthy=False Terminal 1 is not running, or died. Start the Worker.
backend=sapi Kokoro did not load. Run python -m app.speech.fetch_models.
could not open the microphone The Windows toggle in step 2 is off.
wake=off openwakeword is missing. Push-to-talk still works.

Then say "hey Jarvis", wait for the tone, and ask "where am I?".

After pulling new code, stop the client with Ctrl+C and start it again. The Worker reloads on its own.

Voices

By default Open Jarvis speaks with Kokoro, an 82M-parameter neural voice that runs locally on the CPU. The default voice is bm_george, a British male voice chosen for the calm, clipped register people recognise as an AI assistant. Other British male options are bm_daniel, bm_lewis, and bm_fable; the full list is in Kokoro's VOICES.md. Kokoro is Apache 2.0, so there is nothing to license.

To be clear about what this is: it is a British AI-assistant voice, not a clone of the actor who played JARVIS. Cloning a real person's voice from film audio is what right-of-publicity statutes like Tennessee's ELVIS Act exist to stop, and it is not something you want in a public demo.

Set voice_backend to choose:

Value What speaks
kokoro The local neural voice. The default.
auto NVDA if it is running, otherwise the Windows voices.
nvda NVDA only, using your own voice and rate.
sapi The Windows voices.

Every choice falls back rather than going silent: if the Kokoro model files are missing, Open Jarvis logs it and uses the Windows voice.

For Indian English and Hindi through SAPI, add English (India) and Hindi with speech in Windows Settings under Time and Language, then Language and region. The voices are Ravi and Heera for English, Hemant and Kalpana for Hindi.

Settings

Settings live in app/data/settings.json, created on first run. Every key is optional.

Key Default Meaning
worker_url http://127.0.0.1:8787 Where the Worker runs. Also settable with the WORKER_URL environment variable.
privacy_level 1 1 full, 2 text only, 3 offline
language en-IN Answer language hint. Open Jarvis answers in the language you speak.
speech_rate 0 Windows voice speed, -10 slow to 10 fast. Ignored when speaking through NVDA.
ptt_hotkey ctrl+alt+space Push to talk
curtain_hotkey ctrl+alt+c Screen curtain
privacy_hotkey ctrl+alt+p Privacy level
repeat_hotkey ctrl+alt+r Repeat
stop_hotkey ctrl+alt+s Stop
yes_hotkey ctrl+alt+y Confirm
no_hotkey ctrl+alt+n Decline
max_image_width 1024 Screenshot width in pixels before sending
voice_backend kokoro kokoro, auto, nvda, or sapi. See Voices above.
kokoro_voice bm_george Which Kokoro voice speaks
kokoro_speed 1.0 Speaking rate, 0.5 to 2.0
kokoro_model_path blank Override the path to kokoro-v1.0.onnx
kokoro_voices_path blank Override the path to voices-v1.0.bin
wake_word_enabled true Listen for "hey Jarvis". Set false for push-to-talk only.
wake_word hey_jarvis Any openWakeWord model name
wake_word_threshold 0.5 0.1 to 0.99. Lower catches more, raises false wakes.

Hotkeys are written as lowercase key names joined with plus. Do not use Insert or Caps Lock; Open Jarvis warns at startup if a hotkey clashes with a screen reader.

Where things are stored

Everything is under app/data/, which is not committed:

  • settings.json your settings
  • openjarvis.sqlite recent turns, remembered labels for unlabeled controls
  • openjarvis.log what happened, including every action and its result. Screen content is never logged.
  • device_id a random id so the Worker can apply a daily limit per install

Layout

app/        Windows client (Python 3.12)
  capture/  UI Automation tree, screenshot, screen curtain, privacy filter, OCR
  speech/   microphone, speech out (NVDA or Windows voices), earcons
  brain/    Worker client, action runner, orchestrator, system prompt
  store/    local SQLite for settings, turns, remembered labels
worker/     Cloudflare Worker that holds the model keys and talks to the model
sandbox/    safe practice pages
tests/      pytest, no hardware needed by default
bench/      latency measurements and end-to-end checks

Tests and checks

pytest                       # pure logic, fast, no hardware
pytest -m hardware           # needs a display, mic, speakers
python bench/latency.py      # capture, screenshot, curtain timings against the budgets
python bench/e2e_voice.py    # synthesised speech through capture, Worker, and actions on Notepad
python bench/e2e_task.py "open youtube and search and play a new hindi song"

Worker tests: cd worker && npm test.

Troubleshooting

  • "I do not know that action." The client is older than the Worker. Restart python -m app.main.
  • "I cannot reach the tutor service." The Worker is not running or worker_url is wrong. Start it with npm run dev in worker/.
  • "I did not catch that." The recording was under a third of a second or silent. Hold the key a little longer and speak.
  • Hotkeys work inside apps but not on the desktop or home screen. Some manufacturer tools, for example MSI Dragon Center Voice Control, keep an invisible administrator-level window alive. When you minimise everything, Windows puts that window in front, and Windows does not let a normal program's keyboard hook see keystrokes headed to an administrator window. Fix: start Open Jarvis with run_admin.cmd (runs it as administrator), or stop that tool's service. Open Jarvis says a short note at startup when it is not running as administrator.
  • Nothing happens on the hotkey anywhere. Another program running as administrator has the focus. Use run_admin.cmd, or change the hotkey in settings.
  • "I could not open the microphone", or the log says wake word could not open the microphone. Windows has two separate microphone switches and the second one is easy to miss. Open Settings, Privacy and security, Microphone, and turn on "Let desktop apps access your microphone". The toggle above it can read "Allow" while this one is off, and desktop apps like Open Jarvis are blocked by that one. Every audio backend fails with PaErrorCode -9999 until it is on.
  • Open Jarvis does not react to "hey Jarvis". Check the log for wake word listening. If it says the wake word is off, run python -m app.speech.fetch_models. If it is listening but never fires, lower wake_word_threshold to about 0.35. The phrase on its own, "Jarvis", also works but is missed more often.
  • Open Jarvis wakes up on its own. Raise wake_word_threshold toward 0.7.
  • It cuts me off mid-question. It ends the turn after about a second of quiet. Speak without long pauses, or set wake_word_enabled to false and hold the key instead.
  • The voice is the old robotic Windows one. The Kokoro model files are missing or failed to load; the log line starting Kokoro voice unavailable says why. Run python -m app.speech.fetch_models.
  • Slow answers. Chrome with many tabs produces a large tree. Close tabs you do not need, or use privacy level two so no picture is sent.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages