- Good fit for a multi-stage capture/encode/transport pipeline
- Works well with
tokio,windows, andffmpeg-next - Keeps the host codebase native and low-level without switching to C++
Rejected: C++ (more binding and build overhead for this repo shape), C# (wrong fit for the native graphics and transport path).
Direct access to VideoToolbox, Metal, Network, and UIKit/SwiftUI.
Rejected: React Native / Flutter. Wrong fit for this decode/render stack.
The practical Windows desktop-capture API. The pipeline does one full-frame
copy (staging-texture readback into a recycled Arc buffer) and composites
the cursor on the CPU. Dirty-rect metadata is queried but not yet used for
partial encode. A zero-copy GPU path (capture texture straight into the
encoder) remains future work; measure first, because after the v0.2.0
hot-path work the readback is no longer the dominant cost at 1080p60.
The host detects the GPU vendor via DXGI adapter enumeration and walks a
vendor-preferred chain (NVENC / AMF / QSV, then libx264) with per-encoder
low-latency options. FFmpeg is pinned to 7.1 everywhere: the Windows DLLs,
CI's downloaded SDK, and Homebrew ffmpeg@7. The bytes tested are the bytes
shipped. Upgrading to 8.x is deliberately a separate change now that the E2E
harness exists to validate it.
AMF is the fragile path (startup keyframes without parameter sets, strict VideoToolbox level requirements). Its guards were developed against real hardware and get preserved verbatim through refactors.
Baseline H.264 is the simplest compatibility target for VideoToolbox startup. HEVC ships behind a host setting ("Prefer HEVC") until it has been proven on each hardware encoder. Negotiation requires both the setting and the client's advertised decode capability, and any HEVC open failure falls back to H.264 silently.
Hardware encoders ignore bitrate pokes on an open context, so every real
change (ABR rung, slider, codec switch) reopens the encoder session (50 to
200 ms, same stream epoch) and forces an IDR. The old per-frame
apply_bitrate call, which silently did nothing on NVENC and AMF, is gone.
The GUI value is now the ABR ceiling and is labeled accordingly.
v2 replaced the v1 format (FlatBuffers FramePacket, a 16-byte fragment
header, and a fire-and-forget ETERNALHELLO) as a clean break. Both sides
ship together, and each side recognizes the other's legacy traffic well
enough to say "update the other half". The break was the point: v1 had no
version field, no session identity, and no back channel, so compatible
evolution wasn't possible.
- FlatBuffers removed. The only dynamic field was the frame payload itself; width, height, and codec belong in the control plane (STREAM_CONFIG). So media is a fixed 32-byte header plus raw Annex B. One less serialization layer on the per-frame hot path, one less parser to fuzz.
- No app-layer checksum. UDP's checksum plus magic/version/session checks and strict length validation catch stray and truncated datagrams. A corrupted-but-valid datagram costs at most an artifact until the next keyframe, which the recovery path requests anyway. A CRC would tax every packet to protect against the rarest failure with the mildest consequence.
- FEC deferred. Consumer-WiFi loss is bursty, which single-XOR parity handles poorly. Pacing, keyframe recovery, and ABR carry v0.2.0; packet types and flag bits stay reserved for FEC if measurement ever justifies it.
- Client liveness is its receiver reports (500 ms cadence) rather than a dedicated heartbeat message. The reports must flow anyway to drive ABR.
Rejected: TCP (head-of-line blocking), WebRTC (too heavy), RTP/RTCP (we would use a fraction of it and still need custom extensions for input and config).
The host owns the ladder because it owns the encoder; the client just reports what it sees. Loss or keyframe-request pressure steps down, sustained clean reports step up, and the GUI slider caps the ladder.
The host advertises _eternaldisplay._udp (TXT: version, proto=2,
platform), re-upserting every 60 s. The v0.1.x refresh unregistered
first, which created a periodic discovery hole. Exit sends goodbye packets.
Multicast-filtering networks still exist, so manual IP and the QR code stay
first-class.
The extended display drives the bundled
VirtualDrivers/Virtual-Display-Driver.
Its MIT license is verified, and its license text ships in the installer
next to the driver. The host doesn't run elevated, so the installer
registers two SYSTEM scheduled tasks (enable/disable) that the host
triggers via schtasks, avoiding a per-toggle UAC prompt.
- Enabled only while an iPad is connected. Disabled on exit, target change, startup, panic, and 10 s after client loss (the v2 liveness signal finally made idle teardown possible; the grace period lets a quick reconnect keep it).
- The iPad mirrors the primary display while the virtual display starts, and the stream restarts onto it once it attaches. That capture must deliver a real image within 5 s; a display that fails to duplicate or only times out goes back to the mirror and counts as a failed start. The 5 s run from the switch, across any pipeline restart meanwhile. Capture never waits for the driver: one worker thread makes every change, in order, and quitting stops it before its final Disable. The iPad gives up 10 s after the handshake without a frame, which a slow Enable task plus the attach wait could exceed. With the mirror showing meanwhile, the display has 30 s to attach after the Enable instead of 10 s. A start that fails is remembered until the user clicks Try again, restarts the stream or changes the mode, so reconnects mirror instead of toggling the driver again. The host changes the driver at most once per 10 s on its own.
- Before enabling, the host writes
vdd_settings.xmlso the virtual display offers the iPad's native landscape resolution and refresh. There is an opt-out in Settings. The worker also names the main screen's graphics card there just before each Enable, the driver's only way to pick a card, and replaces only a name it may: on a driver Setup installed, an empty ordefaultone, or its own last one. A start that fails, and a second screen that shows no picture, log a display snapshot while the display is still on, with the card the driver rendered on. - A first-party signed display driver remains a long-term goal.
Touches are normalized over the displayed video (letterbox-corrected), which makes the wire format resolution-independent. Tap-vs-drag-vs-scroll disambiguation is a pure state machine: tap clicks on release, a drag commits after slop, and the Pencil presses immediately because ink can't wait out a disambiguation window. Edges are sent twice with one event id and deduped host-side, which buys loss tolerance without retransmit machinery. Injection maps through the captured output's desktop rectangle, so multi-monitor and virtual-display layouts land clicks on the right screen.
- One version, single-sourced from
host/Cargo.toml(env!into the banner and the mDNS TXT), matched by the iOSMARKETING_VERSION. - CI builds releases from a
v*tag: pinned FFmpeg, and the bundled driver pinned by tag AND by the asset's SHA-256 with a hard fail on mismatch. (Upstream ships the setup wrapper unsigned — the driver files inside are the signed part — so exact-bytes pinning is the supply-chain gate, and the workflow reports the Authenticode status informationally in case upstream starts signing.) The installer's SHA-256 is published in the release body, where the website reads it.
The existing prefix, session-id gate, nonce replay and codec behavior stay intact. NACK, audio, keyboard and pairing additions use capability bits and append-only control tails. Old parsers can ignore optional tails; matching release builds are still the supported combination. v0.1 used a different protocol and cannot stream with v2.
A bounded NACK history repairs the specific lost fragments within a short delay. This avoids multiplying WiFi fragment loss into keyframe storms. FEC remains reserved: it would spend bandwidth on clean links and needs separate device measurements. Keyframe recovery remains available when repair expires.
The first repair window held at most three frames for 8–25 ms and asked for a frame's missing fragments once, retrying once. A physical iPad on the LAN, with the PC wired, showed why that failed on real WiFi even without injected loss. Fragments arrived a few milliseconds out of order, so most repairs were filled by the late original. Stalls released several frames at once, and the three-frame bound evicted frames milliseconds before their last fragment. A second loss in the same frame waited for the retry. A packet capture on the PC showed the iPad's repair requests sometimes arriving about 100 ms late, while the host answered each within 0.1 ms.
The iPad now holds up to eight frames, requests each missing fragment as soon as it is seen, and pauses the 25 ms deadline while media or repairs stop flowing, by at most 100 ms per frame. The host resends a fragment again after 5 ms rather than 20 ms, so a retry can replace a lost resend. On the physical iPad over WiFi with 3% injected loss and 1% reordering, keyframe requests fell from 5–22 to 0–1 per 20 s; the clean and 40 Mbps burst rows had no drops. A frame that is truly lost waits up to 100 ms longer before its keyframe request when nothing else is flowing. Marking the socket as interactive voice did not change the request delay, so the default service class stays.
WiFi also delivered some one-datagram frames after their successor. Handing the successor to the decoder first made the HEVC decoder reject the late frame and every frame that referred to it. The same bitstream decoded cleanly on the iPad offline and over USB. A complete frame that skips a sequence number now waits up to half a frame period for the missing one; keyframes do not wait. When a stall expires several frames together, the iPad sends one keyframe request per 500 ms, the host's own grant interval, instead of one per frame. With both changes the 3% loss row passed three runs in a row with at most one drop and one keyframe request each.
A frame with more than 64 missing fragments used to request a keyframe at once,
because one NACK names at most 64. At 40 Mbps a 3440×1440 frame is 72
fragments, so over Tailscale a short stall made that rule fire, and 13 of 15
such frames then completed from late or repaired fragments. The iPad now
requests a wide gap in several NACKs and asks for a keyframe only when a frame
is dropped. Evidence: EternalMonitor-Handoff/evidence/rc-hardware-20260924/.
Apple's installed device service handles the cable/trust relationship. EMLINK wraps existing v2 datagrams, so the session, pairing, media and input machinery stay shared with UDP. The iPad listener is loopback-only. Host frame queues are bounded and drop whole access units with an IDR request. On 2026-09-23, the reference PC streamed a moving 2732×2048 extended desktop to the physical iPad for 642.9 seconds at 57.2 decoded FPS with zero drops. Rear USB-C with a C-to-C data cable also charged the battery during that run. Timed cable takeover, fallback and first-install trust still require their own device checks.
The AudioToolbox experiment created a converter but produced 840 frames for the
first 960-frame packet and no useful concealment from a missing packet. The
source-built, pinned libopus package gives one tested 960-frame/PLC path. The
FFmpeg low-delay encoder uses CELT, which has no SILK in-band FEC; unavailable DTX
options are not reported as active. Compact reset silence packets cover quiet
periods. See docs/audio-codec.md for the probes, pins and license notices.
A CSPRNG host token, rotating six-digit code, IP cooldown and Keychain persistence prevent an unauthenticated LAN peer from casually taking control. USB trusts physical access. This model does not hide the code, token or stream from a network observer. Adding encryption requires a separate authenticated key-exchange design; the current app is for trusted local networks.
The bounded BSD receive path remains selectable for device comparisons. Local 1440p/40 Mbps burst runs reached about 60 fps with zero drops on both paths, so there was no measured gain to justify changing the existing default. Software simulator results do not settle device behavior.
The installer owns privileged driver setup. A normal host launch needs read and execute access to the SYSTEM enable/disable tasks, not a UAC prompt for every connection. Their ACL grants that access to BUILTIN\Users; the host reports the actual scheduler error. The 2026-09-23 campaign verified these tasks from a Limited user token, including attach, disconnect and host-exit cleanup. The installed host also attaches the physical iPad's extended display without elevating the host. The final campaign on main remains required.
Keep YUV420P as the default. On the reference Windows PC on 2026-09-23, NVIDIA H.264 accepted BGRA for 601.347 seconds at 58.27 decoded fps. The four quadrant mean channel errors against YUV420P were 0, 0, 0.33, and 1 out of 255, below the 12/255 limit.
The pinned FFmpeg SDK's AMD H.264 encoder did not advertise BGRA or BGR0
input. The explicit BGRA run therefore fell back to libx264 and failed the
AMD hardware assertion. Its ten minutes of software streaming do not count
as an AMD BGRA pass. AMD YUV420P passed at 58.50 decoded fps with no dropped
frames, and the SDK decoded the captured 120-packet stream without errors.
Auto and BGRA remain opt-in because the required native BGRA gate did not
pass on both GPUs. Private campaign evidence is in
EternalMonitor-Handoff/evidence/windows-colors-a0823cc/.
NVIDIA's encoders take BGRA under Encoder input's new Automatic. With YUV420P input, the Windows host copied every captured frame and converted it with single-threaded swscale before NVENC. At an iPad Pro 12.9-inch's 2732x2048 Second screen that took about 10 ms of each 16.7 ms frame on the development Mac, and about 9% of captured frames never reached the encoder on the reference PC. With BGRA, NVENC converts on the graphics card. NVIDIA BGRA already passed the 2026-09-23 comparison above. AMD, Quick Sync and x264 keep YUV420P (NV12 on Quick Sync), because AMD's encoder takes no BGRA in the pinned SDK. A BGRA open that fails opens YUV420P at once. YUV420, Auto and BGRA stay available, and a picked one wins.
Automatic applies to fresh settings and to a pick. Every earlier version saved
yuv420 whether or not anyone picked it, so a saved yuv420 can't tell a
default from a choice made to fix wrong colours. Working PCs keep today's
behaviour, so settings from an earlier version keep the input they name, and
Troubleshooting says Automatic is faster on NVIDIA. Automatic is saved as
yuv420 with encoder_input_automatic. After a downgrade, 0.3.2 and 0.3.3
ignore the marker and stream their default YUV420, instead of resetting every
setting or reading Auto, which gives BGRA to any encoder that advertises it,
Quick Sync HEVC included, with no fallback. Their save drops the marker, so
the PC stays on YUV420 after the next upgrade. A picked Auto is saved as plain
auto.
GitHub's free three-vCPU macOS VM cannot run the software x264/x265 encoder and the simulator's software decoder at 55 FPS together. In retained runs the host itself encoded 33-52 FPS there while the app decoded every frame it received, and only about one hosted streaming job in six passed. Hosted CI therefore records simulator FPS but still fails on missing frames, drops, repairs, pixels, resolution, link and audio checks. The 55 FPS gate is enforced by the full local matrix and the 30-minute soak on the Apple silicon development Mac, and each phase PR records those results.
The TestFlight job always compiles an unsigned Release archive for PRs and missing secrets. Signed runs use a team API key and an optional temporary distribution keychain. Build numbers increase with workflow runs; tags must match the marketing version. External-group setup, Beta App Review and the public link remain manual App Store Connect steps. A dry archive is not an installable TestFlight build.
A Wi-Fi uplink can stall for seconds while the downlink still works: the iPad keeps decoding, but the PC hears nothing from it. Ending the session 3 s into such a silence cut the picture of an iPad that was fine, and because the host said nothing, the iPad only reconnected once its own check gave up, about 8 s later.
After 3 s of silence (LIVENESS_TIMEOUT), a client that had a picture now
keeps its session and its stream: video, audio and heartbeats go on. Its held
keys, mouse buttons and pen are released then, and another iPad may take the
slot, as when the session ended. Anything from the client before 10 s
(HOLD_LIMIT) carries the session on, with one IDR under the usual keyframe
limit. At 10 s, or when another iPad takes the slot, the session ends with
end=liveness and the host sends BYE(error). Late packets of that session get
it again, at most once a second for a minute. Every 0.3.x iPad reconnects at
once on error, and ignores a BYE for a session id that isn't its own.
- HELLO_ACK still advertises 3000 ms. Every iPad's own check of the PC runs on it, so a larger value would slow down noticing a PC that is really gone.
- A client that never counted a frame still ends at 3 s, so startup failures
keep their
no_videodiagnostics. - The second screen counts a held session as gone, so the virtual display still goes 13 s after the iPad's last packet.
- The stream goes on during the hold rather than pausing, because a pause would freeze an iPad whose downlink is fine. For an iPad that is really gone, that costs up to 7 s of video sent to nowhere.
- Input that waited in a stalled uplink is dropped rather than landing seconds
late, perhaps in another window. Every 0.3.x iPad stamps input and pings with
one monotonic clock (
client_time_us,t1_us). The session keeps the smallest difference between host time and that stamp: the clocks' offset plus the quickest trip, allowed to rise 1 ms a second so a slow iPad clock never looks late. Input over 1 s (INPUT_LATE) past it is late. The relay drops it, except a release of a key, button or Pencil contact the PC still holds, which lands where the PC last saw it. A dropped press also drops its release, and a Pencil stroke whose start was dropped is dropped to its end. Unstamped input is never late. - Logs:
Client silent — holding the session(WARN),Client heard again — session kept(INFO, withsilent_ms,held_msandheard), andDropped input that arrived late(INFO,late_inputandmax_late_ms, a second after a burst of late input ends). The expiry WARN addssilent_ms,held_ms,last_heardand the session'slate_input. Thesession_summaryline is unchanged. - With auto-reconnect off, an iPad whose session ends this way says the PC
ended the session, and its own summary says
end=goodbye.
Rejected: a longer liveness timeout, because held input and the device slot would wait for it too, and pausing the stream during the hold.
- Physical iPad and full reference-PC campaign results: tracked in the hardware runbook.
- FEC, encryption, zero-copy GPU capture and dirty-rectangle encoding.
- Native macOS screen capture via ScreenCaptureKit; macOS currently supplies synthetic capture.
- A first-party signed display driver.
- Public App Store distribution; testers use TestFlight or source builds.