Use IOSurface on macOS/iOS

[madsmtm]

Make Buffer<'_> be backed by a shared memory buffer that can be shared directly with the compositor. This makes the CoreGraphics backend zero-copy (QuartzCore was copying from the CGImageProvider internally before when committing the transaction).

This also implements triple buffering of the surface, to avoid creating a bunch of unnecessary buffers. The compositor seems to want to work on two buffers at the same time? A bit unsure why, but I know that we do get tearing if we try to touch things while it's working on it.

Fixes #83.

This requires #321, because IOSurface does not support RGBX (so the alpha channel must be opaque to render correctly).

Tested on:

• macOS 15.7.3 M2
• macOS 10.12.6 Intel
• iOS simulator

Replaces #95 and #96.

TODO:

• Figure out what to do about color spaces.
• Add a way to control write-combined memory: Allow reading from buffers #320.

[madsmtm]

The way we wait for the compositor to stop using the buffer(s) here is kinda bad, but that can be resolved later, and shouldn't be a problem for applications that either:

1. Only re-render when something changes.
2. Do proper frame-pacing (Winit doesn't yet provide that though).

[ids1024]

When I run the animation example, it seems to sometimes work, but sometimes get stuck in this loop after the three buffers have been allocated.

[ids1024]

Hm. When I try this on an M1 Mac Mini running Tahoe 26.2, I'm not seeing it work correctly. Examples create windows with white contents, except it becomes partly visible if I drag another window over it. But that doesn't show up in a screenshot, which just shows the window white.

[madsmtm]

It requires #321 (the alpha mode is AlphaMode::Opaque), if you want to test things you can do:

diff --git a/src/pixel.rs b/src/pixel.rs
index bcc223d..eb08f30 100644
--- a/src/pixel.rs
+++ b/src/pixel.rs
@@ -101,7 +101,7 @@ impl Pixel {
     /// ```
     pub const fn new_rgb(r: u8, g: u8, b: u8) -> Self {
         // FIXME(madsmtm): Change alpha to `0xff` once we support transparency.
-        Self { r, g, b, a: 0x00 }
+        Self { r, g, b, a: 0xff }
     }
 
     /// Create a new pixel from a blue, a green and a red component.

[ids1024]

Ah. Is there any reason we have to wait for #321 to change this to 0xff? I guess we specified that the alpha component should be 0 before #289 as well, but there aren't really platforms where that's necessary.

Though hopefully we can get both these PRs merged soon.

[madsmtm]

Idk., I just preferred to wait for #321 (and I won't merge this one before that)

[madsmtm]

Comparing this against the current master, it seems that IOSurface doesn't respect the color space we request with kIOSurfaceColorSpace? It seems to instead render with the display's current color space (usually Display P3), as-if we were doing something like the following each time the display mode changes:

let display_id = match self.window_handle.window_handle().unwrap().as_raw() {
    RawWindowHandle::AppKit(handle) => {
        let view: &NSView = unsafe { handle.ns_view.cast().as_ref() };
        let window = view.window().unwrap();
        let screen = window.screen().unwrap();
        let desc = screen.deviceDescription()
            .objectForKey(&NSString::from_str("NSScreenNumber"))
            .unwrap()
            .downcast::<NSNumber>()
            .unwrap()
            .unsignedIntValue()
    }
    _ => todo!(),
};
surface.color_space = CGDisplayCopyColorSpace(display_id);

Unsure how QuartzCore handles this, does it re-render in the client application, or does it somehow tell the compositor what the color space is supposed to be? Will have to reverse-engineer.

(Not that we're really specified what the color space is supposed to be in Softbuffer, see also #15, but fruit.jpg currently renders the same as it does with the Preview app, and it doesn't after this PR).

[nicoburns]

This is much much faster than main for me (like 10ms+ faster, softbuffer time reduction by 90%+). However, I'm running into issues with the size of the surface geting out of sync with the size I recieved from Winit resize events. Not sure which one is stale, but the sizes don't match triggering an assert in my code. I tried changing the assert to be an early return (so skip the frame if out of sync), but it seems that it stays out of sync on subsequent frames (even if resizing has stopped) so that doesn't help much.

[nicoburns]

The memory usage of this is looking to me (doesn't have the same problem that an earlier PR had)

[madsmtm]

However, I'm running into issues with the size of the surface geting out of sync with the size I recieved from Winit resize events. Not sure which one is stale, but the sizes don't match triggering an assert in my code

Probably a Winit bug, as discussed in #275 (comment), Winit sends two redraw events when resizing? And I wouldn't be surprised if one of them had different window.surface_size() values.

But if you use Buffer::width/height/byte_stride when rendering, you should always have the correct size for the buffer (even if the window might be slightly differently sized at that point).

[nicoburns]

I wouldn't be too concerned if there was just a frame (of few) lag on the sizes converging. What's more concerning to me is that even after resizing stops, the buffer/window sizes never end up matching again once the window has been resized once (they will occasionally match up again during resizing if you resize a lot, but I have been unable to get it to finish a resize in a matching state).

[madsmtm]

And that's just with this PR?

[nicoburns]

@madsmtm I looked into this again today. And I have good news: the issue here was just that I wasn't accounting for the buffer's stride. Accounting for that now, and this is working great.

It does seem to be slightly slower to copy pixels a row at a time to account for the stride compared to simple copy of the entire buffer (I'm currently rendering into an intermediate Vec<u8> buffer and then copying into the IOSurface as vello_cpu is not currently capable of rendering into a buffer with a stride). But this is still much much faster than main (and now working reliably!).

[madsmtm]

It does seem to be slightly slower to copy pixels a row at a time to account for the stride compared to simple copy of the entire buffer

I can imagine, but nothing we can do to get around that, stride is fundamental to the problem.

(CC linebender/vello#1382)

[madsmtm]

To make it a bit clearer what my plans are, I'll mark this as a draft, since I haven't found a good way to handle the color space issues noted in #329 (comment).

[nicoburns]

I now have stride support implemented in vello_cpu here: linebender/vello#1597

[madsmtm]

Tested color spaces a bit more, if you're rendering with CGColorSpace::new_device_rgb() with CoreGraphics (as we are on master), Buffer::present lies between 1300µs and 4200µs.

But if you set the color space to the current screen's color space (such as P3, retrieved as described in #329 (comment)), the presentation time drops to between 200µs and 700µs.

In comparison, presentation times when using just IOSurface lies between 50µs and 150µs.

---

This leads me to believe that when rendering with the current screen's color space, CoreGraphics has a fast path where it copies from the provided buffer to an internal IOSurface (or similar DMA-backed buffer) without any sort of color space conversion that it otherwise does in the normal sRGB case.

---

Something interesting to test would be to see what happens when you drag a window rendered with one screen's color space (such that it uses the above fast path) onto a different screen while the application is blocked (such that CoreGraphics doesn't get a chance to submit further work to the compositor)?

A theory is that if the window is rendered in the wrong color space on the new screen, then that would be a hint that the compositor simply doesn't support color space conversions, but if not, then that would hint that it does, and that there might be a way to make use of it?

[madsmtm]

So, turns out that I was wrong about IOSurface / the compositor not supporting color space conversions!

Looking at WebKit sources with @jrmoulton at RustWeek today (thanks for the help!), I realise now that there's a difference between "properties" and "values" on IOSurfaces, namely that properties are set at creation and are immutable, values are mutable and set with IOSurfaceSetValue.

Using this instead, I've now managed to get the same visual result as with current master. Will clean up the PR at some point, but then it should probably be possible to land! 🎉