Skip to content

feat(linux): auto-sample X11 cursor telemetry when portal metadata is missing - #1

Open
Jasonsu22 wants to merge 344 commits into
mainfrom
cursor/x11-cursor-telemetry-fallback-19b6
Open

Jasonsu22 wants to merge 344 commits into
mainfrom
cursor/x11-cursor-telemetry-fallback-19b6

Conversation

@Jasonsu22

@Jasonsu22 Jasonsu22 commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

On X11 + Xvfb hosts the ScreenCast portal often has no METADATA cursor mode, so editable-cursor takes used to write no sidecar. This PR samples the pointer automatically during record (Electron screen.getCursorScreenPoint(), then xdotool / xinput / xev / libinput) and writes version-2 <video>.cursor.json next to the take. openscreen export --auto-zoom then works with no extra import step.

Also adds openscreen cursor import for an external ffmpeg/x11grab CSV track.

Follow-up on this branch addresses CodeRabbit’s review of the first revision: discard stops the X11 sampler, fallback samples are compacted by pause ranges, the sampler clock starts at capture start, cursor import CSV/write guards are stricter, XI2 rest no longer accumulates, click watchers fail over on child exit, and the browser fallback keeps the source it just picked.

Related issue

Fixes getopenscreen#1415

Type of change

  • Bug fix
  • Feature
  • Enhancement
  • Documentation
  • Refactor / maintenance
  • Performance
  • Security

Release impact

  • Patch
  • Minor
  • Major / breaking change
  • No release note needed

Desktop impact

  • Windows
  • macOS
  • Linux
  • Installer / packaging
  • Not platform-specific

Repro host (from getopenscreen#1415)

Item Value
Host Grok Bot cloud Linux (grok-bot-vm-316768464)
OS Debian 13 (trixie), kernel 6.12.94+ x86_64
Session X11 DISPLAY=:2, Xvfb 1920×1200×24
Portal ScreenCast METADATA unavailable (AvailableSourceTypes=0)
GPU Mesa llvmpipe (CPU export fallback)
OpenScreen tested Linux AppImage 2.1.2

Behavior

  • Auto path: X11 (incl. Xvfb) — PipeWire fail → Electron/xdotool/xinput/xev/libinput; native Linux capture with empty cursor track fills sidecar via parallel X11 sampler
  • CLI: openscreen cursor import --csv mouse.csv --clicks clicks.txt --video take.mp4
  • Env: OPENSCREEN_DISABLE_X11_CURSOR=1 / OPENSCREEN_FORCE_X11_CURSOR=1

Testing

  • Unit tests for sampler / sidecar import / CLI parser / factory (vitest)
  • tsc --noEmit and tsc -p tsconfig.test.json --noEmit
  • Targeted vitest after the CodeRabbit follow-up
  • Maintainer smoke on a normal Linux desktop (should be no-op when METADATA works)
  • Optional: X11/Xvfb agent host record → sidecar present → export --auto-zoom

Notes

Upstream PR (same branch): getopenscreen#1416

Open in Web Open in Cursor 

EtienneLescot and others added 28 commits November 7, 2026 15:20
The rules Rusuban's agents follow on this repo, read from the default branch: what they do alone
and what comes to the maintainer, what merges alone and how this repo merges, review rules, and
how they write on GitHub. locks.json holds what the server checks without the model: supply-chain
paths, size limits, the zones that need a rendering and performance proof, CodeRabbit required, and
the Nix build and notification workflows ignored.
…en, contributor PRs kept

From the review: *.spec.* files are tests; electron/native-bridge/ is capture code and needs the
rendering proof; a fork that refuses maintainer edits gets a fallback that keeps the contributor's
commits; a worker finishes a contributor's PR on their branch, never in a competing one.
…ead before they are carried

- Every card goes to the owner.
- A review cancelled or skipped by CodeRabbit's hourly limit: `@coderabbitai review` once per head,
  then wait.
- A contributor's commits carried to a new PR are read first: one touching `.github/`, `scripts/`
  or the build is never pushed here, where its workflows would run with the repo's secrets; the new
  PR carries the closing reference; the original's head is read again before it merges.
The HUD grants HUD_GROWTH_RESERVE beyond its required size on each axis
(+240px width, +120px height) as hysteresis against native resizes. On
Windows and macOS that overhang is click-through. On Linux the HUD is not
click-through, and a Wayland compositor treats the whole toplevel as the
window's allocation, so the reserve is a transparent rectangle around the
bar that swallows desktop clicks.

hudGrowthReserve(platform) returns 0 on Linux, and the renderer passes it
into computeHudWindowSize and uses the same value for the shrink threshold,
so granted equals required there while other platforms keep the reserve.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The HUD cannot be click-through on Linux, so every pixel of transparent
window reserve is a dead zone over the desktop. Size the window to the
measured stack instead: the bar alone (plus a floor for portalled
tooltips) while collapsed, and the open panel's measured height while one
is open, shrinking back when it closes.

The window height now also counts the visible toast stack, which the
growth reserve used to cover. Without it, a Linux window that hugs the bar
cuts the toast off. The same height feeds needsResize, so a toast
appearing or leaving costs one resize.

Linux also lifts the fixed min/max size hints (resizable: false) around
setBounds, since Mutter otherwise ignores the resize, and starts at the
compact bar size instead of the old 820x560 reserve.

Co-Authored-By: Claude Code <noreply@anthropic.com>
- llms.txt: install commands (Homebrew core cask, winget, Linux, nix),
  a note that the old siddharthvaddem tap stops at v1.5.0 and that no
  getopenscreen/tap exists, plus the custom effects doc, the Edit videos
  with Claude page and the Off-Menu post.
- FAQ (8 locales): lead with brew install --cask openscreen; the archived
  README no longer calls this a spin-off, it simply points here.
- README: Homebrew install for macOS; official links list Store, Homebrew,
  and states forks are not official.
Review follow-up for getopenscreen#1328.

- Direct X11 capture only when the ScreenCast portal does not answer on an
  X11 session. The helper decides and reports it as `backend` on `ready`;
  the main process asks it once per process, and never on Wayland.
- Frames come through a server-created MIT-SHM segment instead of the X
  socket, with plain GetImage kept for servers that cannot share memory.
- Window capture reads the root over the window rectangle, clipped to the
  screen: no BadMatch past the edge, and covered areas show what is on screen.
- A non-zero helper exit after `capture-stopped` keeps the finalized file
  again, as on Wayland.
- Clicks made during the countdown are dropped, and the French CLI page
  loses its doubled "Wayland".
@cursor
cursor Bot force-pushed the cursor/x11-cursor-telemetry-fallback-19b6 branch from 4d76cd5 to 5fd254a Compare November 17, 2026 22:49
EtienneLescot and others added 24 commits November 18, 2026 19:23
On the proprietary NVIDIA driver, VA-API cannot map a compositor's tiled
DMA-BUF, so the helper only offered LINEAR to a DMA-BUF-only producer
such as niri, which renders corrupted frames into it there. OBS reads
the same buffers by importing them as an EGL image and reading back
through OpenGL.

When the VA-API importer does not build and EGL can import DMA-BUFs
with modifiers on a hardware device, the DMA-BUF offer now carries the
modifiers that EGL display imports. Each tiled frame is imported with
eglCreateImageKHR on the PipeWire thread, attached to a texture-backed
framebuffer and read back with glReadPixels into an RGBx buffer, so it
reaches the encoder as an ordinary CPU frame. libEGL and libGLESv2 are
dlopen'd, like libpipewire.

The VA-API path, and the shm-first order without it, are unchanged.

Refs getopenscreen#1503
eglChooseConfig defaults EGL_SURFACE_TYPE to EGL_WINDOW_BIT, so a device
display with only pbuffer or stream configs found none and disabled EGL
import. The context never draws to a surface.

Refs getopenscreen#1503
The DMA-BUF offer pairs its modifier list with BGRx, RGBx, BGRA and RGBA,
but the EGL branch only queried XRGB8888. EGL reports modifiers per
format, so a producer could pick a pair the readback cannot import and
every frame would be dropped. Offer the intersection instead.

Refs getopenscreen#1503
github.com/microsoft/onnxruntime returns 404 since 2026-11-17, release
assets included, so every CI run failed at "Stage ONNX Runtime". The
win-x64, win-arm64, linux-x64 and linux-arm64 targets now come from
Microsoft.ML.OnnxRuntime 1.27.1 on api.nuget.org: the same release,
Microsoft-signed, immutable per version, SHA-256 pinned. darwin-arm64
keeps its own build.

The package holds every runtime, so members are named by path, and a
zip is extracted with unzip where tar is GNU tar.

Refs getopenscreen#1625
…mabuf extensions

On a hybrid AMD + NVIDIA Linux laptop, wgpu picks the NVIDIA adapter, which
is opened without VK_KHR_external_memory_fd / VK_EXT_external_memory_dma_buf,
while VaapiEncoder::open succeeds on renderD128 (the AMD iGPU). The export
then called vkGetMemoryFdKHR anyway: ash panicked with "Unable to load
get_memory_fd_khr" in a non-unwinding context and the whole app aborted.

Check the device's enabled extensions first and return None, as the doc
comment already promised, so run_multi_inner falls back to the software
encoder. Adds a regression test that opens a device through the standard
request_device path; without the guard it reproduces the abort.

Fixes getopenscreen#1622

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks the extensions

The round-trip test skipped on any None from create_exportable_staging, so a
guard that wrongly refused a capable device would have passed silently. Check
the device's enabled extensions first, skip only when one is missing, and
otherwise require Some. The check is shared with the regression test through a
small has_external_memory helper.

Addresses the CodeRabbit review on getopenscreen#1624.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t only extensions

A device can enable both external-memory extensions yet refuse to export a
TRANSFER_DST buffer as dmabuf, or offer no HOST_VISIBLE | HOST_COHERENT memory
type for it. create_exportable_staging then returns None, which the round-trip
test would report as a failed guard. The test now checks those capabilities
and keeps its expect once they hold.

Refs getopenscreen#1622
A long fragmented MP4 from the Linux recorder takes over 5 s to reach
loadedmetadata (12.7 s cold for an 11 min take), so the on-load probe
timed out, the asset was marked probed for the session and the editor
stretched the picture. The backfill now waits up to 60 s and retries a
failed probe on a later document change, at most three times.

Refs getopenscreen#1610
A save deferred by waitForDocumentSaves no longer uses up a probe
attempt, and a run started while a batch is still probing no longer
picks up a queued asset of that batch.

Refs getopenscreen#1610
The batch is reserved up front, so a throw from toFileUrl skipped the
cleanup and stranded every asset in it. Catch it per track instead.

Refs getopenscreen#1610
…atches keep both

Two batches waiting on the same pending save each re-read the same document,
and the second save dropped the first one's dimensions while those assets
stayed marked as probed. Chain the wait, re-read and save on a shared promise.

Refs getopenscreen#1610
…audit

The named write helper moved the saveDocument call site out of useTimeline,
which the document write audit tracks by enclosing function. Chain the
write inline instead.

Refs getopenscreen#1610
A probe that succeeded but whose save failed stayed reserved for the
session, so its dimensions stayed missing until the editor remounted.
Release it on a failed save, without counting a probe failure.

Refs getopenscreen#1610
…stall the chain

The backfill batches write one after another, so a save that never settled
held every later batch and kept their assets reserved for the session. Wait
on it through saveWithDeadline and release the batch on a timeout, as on a
failed save.

Refs getopenscreen#1610
…ttach

The driver gated seeks on a flag set by the video's loadeddata event. The
video outlives the driver, so when a media-query change re-attached it the
event had long fired and the bubble froze for good. Gate on readyState
instead, and on HAVE_METADATA so targets issued while a seek runs into an
unbuffered range are kept rather than dropped.
The start funnel caught every failure, showed its toast and resolved
normally, so a caller could only learn of a failed start from recording
state that never flips. startRecording now answers with an outcome:
started, cancelled, or failed with the message the toast showed. The
countdown and restart paths ignore it as before.
A start the recorder had already failed left the hidden runner guessing:
its deadline answered MCP startRecording and `openscreen record` with
"Recording did not start within 30s — see stderr". The runner now fails
at once with the outcome's reason, says when the picker was dismissed,
and appends the system-cursor retry advice only to the cursor-metadata
failure. The deadline remains only as a backstop for a start that
reports success but never confirms; in a race with the camera effect
(getopenscreen#1303) the first error still wins.
FakeRunners can now name a record start's failure. Assert the
startRecording tool error carries that reason, so the forwarding above
the runner stays honest.
ggml turns Metal on by default for every Apple build, so the macOS x64
helper ships libggml-metal and, without --cpu, decodes on the Intel iGPU.
Pass --cpu from the first launch whenever the backend is whispercpp-cpu.

Refs getopenscreen#1594
An explicit binaryPath with no backend is auto; its whispercpp-cpu
default is a label only and must not force --cpu on first launch.

Refs getopenscreen#1594
On X11 (including Xvfb hosts) the ScreenCast portal often offers no
METADATA cursor mode, so native Linux captures in editable-overlay mode
came back with an empty cursor track and no `<video>.cursor.json`.

While a native Linux capture runs in editable-overlay mode, sample the
pointer in parallel through X11 (fresh read-only xdotool positions,
converted to DIP; clicks from xinput test-xi2, with xev / libinput
fallbacks). It never talks to the portal, so there is no second picker.
When the helper's own cursor track is empty, the sampled track is
compacted by the helper's pause ranges, bounded to the video duration and
written as the sidecar. Helper samples are still preferred when present.

Samples are bound to the captured monitor; ambiguous or window geometry
produces no samples rather than a guess. OPENSCREEN_DISABLE_X11_CURSOR=1
turns the sampler off. Browser capture and the cursor import command are
out of scope for this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jasonsu22
Jasonsu22 force-pushed the cursor/x11-cursor-telemetry-fallback-19b6 branch from a16a217 to 4ee57df Compare November 22, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Linux X11/Xvfb: auto-sample cursor telemetry when ScreenCast METADATA is unavailable