Fix absolute mouse position on multi-monitor targets + touch input - #1519
Fix absolute mouse position on multi-monitor targets + touch input#1519imcynic wants to merge 2 commits into
Conversation
…ts and touch clients Absolute mouse coordinates (0..32767) are interpreted by the target OS across the bounding box of the entire desktop, while JetKVM captures a single screen — on multi-monitor targets every click landed far from the video position (jetkvm#1446). Adds an optional device-side mapping (config + getAbsMouseMapping/setAbsMouseMapping RPCs + authenticated /mouse-mapping HTTP endpoints) that remaps coordinates onto the captured screen's rectangle, applied at the single rpcAbsMouseReport choke point, plus a Mouse settings section to configure it. Touch clients (iPadOS Safari): absolute handler now uses clientX/clientY against getBoundingClientRect instead of offsetX/Y (unreliable for touch-synthesized events on WebKit), listens to pointermove instead of mousemove in absolute mode, marks the video touch-action:none so drags emit pointer events instead of scrolling, ignores non-primary touches, and releases buttons on pointercancel.
- Serialize mapping updates and guard the config pointer with a dedicated RWMutex; roll back in-memory state when persisting fails - Bound mapping dimensions (1<<20) against overflow and float precision loss - Return 500 for persistence failures, 400 only for invalid mappings - Capture the pointer on pointerdown in absolute mode so button releases outside the video element are not lost - Clean up the fullscreenchange listener; guard NaN in mapping form inputs - Add on-device unit tests for validate/apply/clamp incl. overflow and boundary cases
|
|
Not signing a CLA, you're welcome to reference and rewrite |
2 participants
Summary
Two related absolute-mouse positioning fixes:
1. Multi-monitor coordinate mapping (fixes #1446). The USB HID absolute pointer (0..32767) is interpreted by the target OS across the bounding box of the entire multi-monitor desktop, while JetKVM captures a single screen — so on multi-monitor targets, clicks land far away from where you click on the video (measured 1700–3000 px off on a 3-monitor Linux target). This adds an optional device-side remap of absolute coordinates onto the captured screen's rectangle within the full desktop:
AbsMouseMappingConfigin the device config (disabled by default, nil-safe, validated: positivity, rect-fits-in-desktop with overflow-safe arithmetic, sane upper bound)rpcAbsMouseReportchoke point, covering both the HID-RPC datachannel path and the legacy JSON-RPC path; invalid/absent config falls back to identity (stock behavior)getAbsMouseMapping/setAbsMouseMappingJSON-RPC methods + authenticatedGET/POST /mouse-mappingHTTP endpoints (the latter lets host-side automation keep the mapping in sync when the target's display layout changes, without a WebRTC session)2. Touch-input handling in absolute mode (iPad/tablets). Relative mode needs pointer lock, which doesn't exist on iPadOS — absolute is the only usable mode there, and it had a position offset plus no drag support:
clientX/clientYagainstgetBoundingClientRect()instead ofe.offsetX/offsetY— offsetX/Y is unreliable for touch-synthesized events on iPadOS/WebKit and caused a large offset for touch clientspointermove(relative mode keepsmousemovefor pointer lock), and the video element getstouch-action: none, so touch drags move the remote mouse instead of scrolling the pagepointercancelreleases buttons;setPointerCaptureon pointerdown so a release outside the video element isn't lost (touch already captures implicitly; this covers mouse/pen)fullscreenchangelistener that was added without cleanupTest plan
mouse_mapping_test.go) for validation, the remap math, and clamping — table-driven, incl. identity/disabled/invalid fallback, boundary (0 / 32767), screen==desktop, overflow and bound cases; run on a physical JetKVM (linux/arm)uilint +tscclean; frontend + release builds greenmake test_e2enot run (requires a resettable test device)Notes: new UI strings are currently in
en.jsononly — happy to run the machine-translate pass or adjust wording per your conventions. Also happy to drop the HTTP endpoints if you'd prefer the RPC surface only.