Skip to content

Add experimental Vulkan GPU rendering and export - #1138

Merged
jonoomph merged 51 commits into
developfrom
vulkan-mvp
Nov 10, 2026
Merged

jonoomph merged 51 commits into
developfrom
vulkan-mvp

Conversation

@jonoomph

@jonoomph jonoomph commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

This adds an experimental GPU pipeline for video decoding, editing, preview, and export. Supported operations keep frames on the GPU, while other operations continue through the CPU path.

  • Hardware decoding: Use FFmpeg Vulkan Video on Linux and Windows, and VideoToolbox shared surfaces on macOS. Prefer hardware AV1 decoding when the device supports it, with software decoding as a fallback.
  • Color handling: Handle 8-bit and 10-bit sources, tagged YUV matrices and color ranges, PQ/HLG HDR tone mapping, and BT.2020-to-BT.709 conversion for SDR editing. Tag exports with the appropriate SD/HD color information.
  • Clip transforms and compositing: Accelerate animated position, scale, rotation, shear, opacity, gravity, and fit/crop/stretch sizing, with source-over blending of overlapping timeline layers.
  • Images and thumbnails: Upload still images, PNGs, and cover art with reusable GPU caching. Resize decoded frames on the GPU, including high-quality bicubic downscaling, and read back thumbnail-sized images when CPU pixels are needed.
  • GPU effects: Accelerate Saturation (overall and per-channel), Crop (offsets and canvas resizing), Blur (regions, radii, and iterations), and Mask (brightness, contrast, inversion, and image replacement).
  • Color grading and LUTs: Accelerate temperature, tint, exposure, contrast, shadows, highlights, saturation, vibrance, mix, color wheels, master/RGB curves, and 1D/3D .cube LUTs, including animated controls.
  • Effect masks: Blend supported effects with still-image and video masks on the GPU, including inversion and drive-amount blending. Generated masks can be uploaded for GPU blending.
  • Video scopes: Calculate luma/RGB histograms, waveforms, vectorscopes, and clipping summaries on the GPU, returning counters without reading back the full preview image.
  • Native Qt preview: Share GPU frames with OpenGL on Linux and Windows and through IOSurface/Metal on macOS, preserving Qt overlays and falling back to CPU display when sharing is unavailable.
  • Hardware export: Convert GPU output to NV12 and pass it to NVENC through a GPU transfer, native Vulkan encoders through encoder-owned frames, or VideoToolbox through shared IOSurfaces. Support depends on the installed FFmpeg build and hardware; VA-API, QSV, and AMF availability is also probed rather than assumed.
  • Device selection and diagnostics: Select a usable GPU automatically or honor an explicit device preference. Expose device/driver details, working encoders, decode/render/encode counters, uploads, readbacks, and fallback reasons through C++/Python APIs.
  • Memory and recovery: Reuse GPU pipelines, upload caches, and frame pools; account for GPU cache memory and release unused pools. Recover from GPU errors and continue on the CPU after device loss.
  • Builds and compatibility: Keep Vulkan optional at build time and load it at runtime when available. Preserve CPU-only builds and the runtime CPU override, update Linux/Windows/macOS FFmpeg and GPU packaging, and bump the library to 1.2.0-dev1 with libopenshot-audio 1.1.0 required. Add integration coverage and CPU/GPU benchmark tools.

Support limits: GPU acceleration depends on the operating system, device, driver, codec, and available sharing features. Other effects and unsupported rendering paths use CPU fallback; GPU output is not guaranteed to be pixel-identical to CPU output. Native Vulkan H.264/HEVC encoding requires a newer FFmpeg build than the packaged FFmpeg 7.1.

Companion application PR: OpenShot/openshot-qt#6242.

Work-in-progress Vulkan MVP, built only with -DENABLE_GPU_FRAME=ON.
CPU builds are unchanged.

- Frames can hold an immutable deferred GPU image and only read back
  CPU pixels when GetImage() is called.
- FFmpegReader decodes through a shared Vulkan device and scales on the GPU.
- Clip transforms and source-over compositing run in a Vulkan compute shader.
- FFmpegWriter hands GPU frames to NVENC through Vulkan/CUDA interop.
- VulkanOpenGLPresenter draws GPU frames in a QOpenGLWidget without readback.
- Timeline and export benchmarks compare CPU and Vulkan builds.
Compile the compute shaders into C headers at build time instead of
loading SPIR-V from absolute build-directory paths, so installed and
packaged GPU builds work.
- OPENSHOT_RENDER_BACKEND=cpu now disables Vulkan decoding at runtime in
  a GPU build, so every frame follows the unchanged CPU path. Output is
  identical to a CPU build.
- A failed GPU export no longer aborts. The writer reads the frame back
  and encodes the rest of the export from CPU pixels.
- Compare clip composite modes against openshot::CompositeType.
Vulkan tests no longer need opt-in environment variables. They detect
a Vulkan device, display and OpenGL interop themselves and print a
"Skipping Vulkan test:" message that CTest reports as skipped, not
passed. The reader cache test uses bundled media by default, and the
renderer test waits for queued delivery on any Qt platform.
Each pipeline now reuses a small ring of command buffers, fences and
descriptor sets. A submit no longer blocks the CPU; later readers wait on
the frames' timeline semaphores, and each slot keeps its frames alive until
the GPU is done with them.
Timeline layers only need the composited result, so draw the transformed
clip straight onto the background. Layers with after-keyframe effects or
timeline effects keep the separate passes.
Freed GPU frames return to FFmpeg pools that never shrink. Keep the
compositor pools in one shared list, so ReleaseVulkanMemory() can drop
them; FFmpeg frees each pool once its last frame is released. Submit
slots no longer hold frame references, since FFmpeg waits on a frame's
semaphores before freeing it.
A decoded frame was held until the audio stream was a full second past it,
so every reader kept 1-2 seconds of decoded images. With GPU decoding that
cost about 0.9 GB of video memory per 4K clip. A 0.25 second margin gives
identical playback audio on 22 test files (AAC, MP3, Vorbis, Opus, FLAC,
AC3, ALAC and PCM) and cuts that to about 0.34 GB.
FFmpeg binds a Vulkan frame to the first CUDA context it is sent to, and
pooled frames were reused by the next export. The second export in a
session then fell back to the CPU, and the third could crash. Release the
pools when a writer opens and closes, and report export and decode
fallbacks as warnings so they show in the console.
FFmpeg 7's MP4 reader returns 0,0,0 orientation, like 8 and 9. The
written angles are still checked directly in the file.
The runner image now has FFmpeg 7.1.3 with Vulkan and newer Vulkan
headers in /opt. The job fails if the system FFmpeg 4.2 is also linked.
FFmpeg uses the first Vulkan device, which is often a software driver
such as llvmpipe, so decoding quietly fell back to the CPU. Try GPUs by
name, discrete first, skip software devices, and log the chosen device.
OPENSHOT_VULKAN_DEVICE overrides the choice.
AMD and Intel exports no longer copy each frame through system memory.
The final NV12 images use linear DMA-BUF memory that VA-API imports
directly, and the VA-API encoder opens on the card that renders. If
sharing fails, the export continues with the old copy path.

New helpers for the UI: ListGpuDevices(), ActiveGpuDevice() and
GpuEncoders(). The Vulkan tests now use the same device choice as the
app, so they no longer skip when a software driver is listed first.
Write native Vulkan NV12 directly into encoder-owned frames, with CPU fallback for unavailable encoders and incompatible surfaces. Keep encoder state per writer and preserve NVENC's separate-plane CUDA transfer.

Add copy-free and fallback regression tests, readback diagnostics, and GPU compilation in the FFmpeg compatibility matrix. Verified on the Radeon 780M with FFmpeg 6.1 and 7.1.3; NVIDIA hardware testing remains outstanding.
Ubuntu 20.04's loader (1.2.131) hands FFmpeg a stub without a driver
function for the video encode queries, and opening h264_vulkan crashes
(FFmpeg's own CLI crashes the same way). Probe the native Vulkan encoders
only with loader 1.3.270 or newer, the oldest verified to fail cleanly.

Use h264_vulkan/hevc_vulkan only when the probe opened them, falling back
to the CPU encoder otherwise, instead of trusting that FFmpeg includes them.

Verified on an RTX 5070 (driver 580) with loaders 1.2.131 and 1.3.270:
Vulkan tests pass, NVENC exports stay on the GPU, and the Vulkan presets
fall back to libx264/libx265.
Keep playback and export counters separate, attribute CPU readbacks, and preserve measurement resets across in-flight work. Cache bounded GPU uploads for image sources so scaling, compositing, and presentation can stay on GPU.
Normalize uploaded artwork into the IOSurface compositor path so CPU fallback uses the synchronized Metal readback bridge. Assert the Mac surface type and retain strict pixel checks with useful RGBA diagnostics.
NVIDIA's Windows driver returns zeros when FFmpeg or compositor frames are
read through an exported alias of their memory, from OpenGL or Vulkan, so
native preview drew black. Copy each frame on the GPU into an exportable
image owned by the presenter, which OpenGL imports once per size, device,
and context, along with the semaphores, shader, and vertex state.

The frame no longer changes queue ownership, which removes the invalid
external acquire on concurrent FFmpeg images. The previous frame's fence is
waited at the next draw instead of finishing GL every frame.
…wn at exit

Maxwell GPUs raise a TEX LAYOUT exception, then a driver reset, when OpenGL
samples an imported optimal-tiling Vulkan image. Copy each frame into an
exported buffer instead and let OpenGL unpack it into its own texture.

The shared Vulkan device and compositor output pools are no longer freed by
static destructors, which ran during DLL unload after the NVIDIA driver had
begun shutting down.
Its static destructor freed cached Vulkan frames during DLL unload, after the
NVIDIA driver had begun shutting down, crashing OpenShot on exit.
Reuse one Vulkan frame pool per image size; a pool per upload leaked
device memory. Share still and cover-art images between frames instead of
deep copying them, so they are uploaded once instead of every frame. Wait
for FFmpeg's upload to finish before freeing the CPU pixels it may read
directly through host-mapped memory.

Name the VkResult in presenter and compositor errors, count failed GPU
uploads (logged every 100th) and stop reporting CPU-decoded cover art as
missing GPU decode.
…ice loss

A failed GPU preview draw now frees cached GPU memory and retries after a
growing pause (0.5 s doubling to 30 s) instead of failing on every frame.
VK_ERROR_DEVICE_LOST marks the device lost for the session: Vulkan
rendering is disabled and open readers reopen with CPU decoding, so
editing continues without a restart.
OPENSHOT_GPU_DEBUG=1 logs centre pixels at each preview handoff step,
the shared buffer setup, OpenGL import errors, and draw geometry, and
saves a crash dump on Windows. OPENSHOT_PRESENTATION_TEST=pattern,
OPENSHOT_PRESENTATION_SYNC=wait and OPENSHOT_PRESENTATION_DEDICATED=0|1
change one part of the handoff each. Vulkan readback failures now log
the frame and device details. Search for GpuDebug to remove.
The RTX 2060 debug logs showed OpenGL reading zeros from the shared
preview buffer while Vulkan wrote the right pixels, even with a fill
pattern and CPU waits. The presenter closed each Win32 handle right
after importing it; keep them open until the shared target is released.
OPENSHOT_PRESENTATION_HANDLE=kmt tries global KMT handles instead.
With OPENSHOT_GPU_DEBUG=1, readbacks log their reason, and thumbnails
and image saves are tagged.
The NVENC converter's ring slots kept their last frames, which were mapped
into CUDA, until process exit. By then CUDA had unloaded, so FFmpeg logged
dozens of failed CUDA cleanup calls after the session ended.
Select FFmpeg's native AV1 decoder on Vulkan devices advertising AV1 decode support, preserving the default decoder for software retries. Add hardware decode and output checks.

Validation: three targeted AV1 tests pass on AMD Radeon 780M with Vulkan video enabled. Packaged FFmpeg builds including libdav1d still need validation.
Keep common image effects and scope analysis on GPU surfaces, reuse pipelines across preview sizes, and retain per-submission color grading parameters. Restore Qt texture bindings after native presentation.

Add coverage for animated effects, image sizing, pipeline reuse, GPU residency, and presentation state. Native presentation passes 219 assertions; broader compositor and CPU checks passed during development. A device-loss failure during GUI seek/edit/layout smoke testing remains unresolved.
Patch FFmpeg 7.1 dependency builds to use staging memory for AMD Linux CPU transfers and correct the staging download copy direction. Preserve GPU-to-GPU video paths.

Detect device loss hidden by FFmpeg transfer errors and prevent preview fallback from reading dead GPU frames. Verify the original reproducer, GUI smoke sequence, 906 compositor assertions, and 222 presentation assertions.
Guard the reader's device-loss check and the writer's Vulkan encoder check so builds without GPU frames compile against FFmpeg versions that predate AV_PIX_FMT_VULKAN.
Give ColorMap a Vulkan path, which ColorGrade's LUT now reuses, so LUTs no longer send grading back to the CPU. Blend effect masks on the GPU for effects that render on Vulkan, and count FrameScope histograms, waveforms and the vectorscope on the GPU, reading back only the counters.

Masks match the CPU byte for byte and scope counts match exactly; LUTs stay within one byte. Failures fall back to the CPU. Add GPU-versus-CPU coverage for LUTs, masks and scopes, plus a masked, LUT-graded clip with scopes that needs no readbacks.
Blend Blur and Saturation drive-amount masks on the GPU with the same exact factor tables as the default blend. Read video masks from the decoder's GPU frames and convert and scale them in one compositor pass, for both the Mask effect and common effect masks. Still masks keep their cached CPU scaling and single upload.

Add coverage for drive-amount masks against the CPU and for 720p video masks with no frame readbacks.
MoltenVK compiles shaders with fast math, which may fuse the mask blend's multiply-add and round one byte differently. Linux and Windows keep the byte-exact check.
- Decode with the source's tagged YUV matrix (BT.601 when untagged, as
  earlier OpenShot exports were written) on CPU and Vulkan.
- Convert HLG and PQ (BT.2020) sources to SDR BT.709 with a shared,
  hue-preserving tone curve (BT.2408 reference white at 203 nits). CPU
  uses a LUT-based converter; Vulkan converts in affine.comp.
- Read P010/P012/P016 and P210/P212/P216 frames directly on Vulkan,
  including 10-bit VideoToolbox surfaces on macOS.
- Tag exports with BT.709 (HD) or BT.601 (SD) and encode with the
  matching matrix, on CPU and Vulkan.
- Add SDR/HLG/PQ color chart fixtures and tests comparing CPU, Vulkan
  and expected values.
- Set OPENSHOT_RENDER_BACKEND with qputenv/qunsetenv; MinGW has no
  setenv/unsetenv.
- Skip charts the hardware decoder declines on every platform.
  VideoToolbox on Intel Macs rejects the 320x192 H.264 chart; the Mac
  10-bit surface path is covered by the synthetic CVPixelBuffer tests.
VideoToolbox on an Intel Mac (UHD 630) refuses 320x192 H.264 with
"VideoToolbox session not available" (FFmpeg requires its hardware
decoder for H.264), while 640x360 and larger decode. Regenerate the
charts at 640x384 and require GPU decoding of them on macOS again.
libopenshot no longer links libvulkan on Linux or Windows. It opens the
Vulkan library on first use and loads only the functions it calls. When
the library or a function is missing (headless servers, old systems
without a GPU driver), OpenShot still starts and renders on the CPU.
macOS keeps linking the bundled MoltenVK.
@jonoomph

jonoomph commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

@codecov

codecov Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.37876% with 106 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.22%. Comparing base (528ebb1) to head (16ccf91).

Files with missing lines Patch % Lines
src/FFmpegWriter.cpp 48.93% 24 Missing ⚠️
src/ColorManagement.cpp 81.41% 21 Missing ⚠️
src/AudioWaveformer.cpp 89.14% 19 Missing ⚠️
src/FFmpegReader.cpp 72.72% 15 Missing ⚠️
src/EffectBase.cpp 74.07% 7 Missing ⚠️
src/Clip.cpp 84.61% 2 Missing ⚠️
src/EffectBase.h 0.00% 2 Missing ⚠️
src/effects/Blur.h 0.00% 2 Missing ⚠️
src/effects/Saturation.h 0.00% 2 Missing ⚠️
tests/ColorManagement.cpp 98.49% 2 Missing ⚠️
... and 10 more
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1138      +/-   ##
===========================================
+ Coverage    74.83%   75.22%   +0.39%     
===========================================
  Files          276      282       +6     
  Lines        38296    39090     +794     
===========================================
+ Hits         28659    29407     +748     
- Misses        9637     9683      +46     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Audio-only readers sized their frame cache from video dimensions, so the
cache never evicted and every insert rescanned every cached frame. Long
waveform reads became quadratic: 100 s for a 2 hour file, now 6 s.

- Size audio-only FFmpegReader caches by audio, and keep the cache of
  readers owned by AudioWaveformer tiny (reads are strictly sequential)
- Add ExtractChannelSamples() to decode the mixed envelope and every
  channel's envelope in a single pass
- Place chunk boundaries exactly, so 44.1 kHz audio no longer drifts
  (16 s late after 2 hours) or loses the end of the file
- Add an ApplyKeyframes() overload taking plain envelopes, interpolating
  keyframes within a frame and fixing a one frame offset
@jonoomph
jonoomph merged commit e3728b2 into develop Nov 10, 2026
23 checks passed
@jonoomph
jonoomph deleted the vulkan-mvp branch November 10, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant