[blas] Implement omatcopy2 for cuBLAS and rocBLAS - #766
Open
zjin-lcf wants to merge 3 commits into
Open
Conversation
omatcopy2 applies an element stride to each matrix, which geam cannot express, so both backends reported it as unimplemented. Add a portable SYCL kernel shared by the two backends: a transposing variant that stages a tile through local memory, so that neither the load nor the store steps by a leading dimension, and a plain strided copy for the nontrans case. Unit strides make omatcopy2 equivalent to omatcopy, but routing that case to geam turns out to cost more than it saves. The vendor libraries have to be driven from a host task and with a stream synchronize, whereas these kernels are ordinary asynchronous SYCL. On gfx950 the kernels beat the geam path by 37-77% even when the caller waits after every call, so the dispatch always uses them. The tile shape is chosen per element size, and separately for unit and non-unit strides: with unit strides both accesses are contiguous and the two phases want equal width, while with real strides widening the stores buys nothing and a taller tile amortises the per-tile overhead. Every variant keeps the tile under 17 KB so that three groups stay resident even on CDNA1-CDNA3, which have a quarter of CDNA4's local memory. Extents were measured on gfx950 over strides 1-4 and sizes from 64 to 8192 square. Tested on gfx950 (MI350X): the omatcopy2 unit tests pass for both the compile-time and run-time dispatch layers, alongside a standalone check of 336 configurations covering both layouts, all three transpose modes, unit and mixed and general strides, and sizes straddling the tile extents.
The tile extents were chosen on gfx950, where a wave is 64 items wide, and the strided table picks 1024-item work-groups for the 8-byte types. That is the whole of an SM's thread budget on NVIDIA and leaves only two groups resident, which measures 4% slower on double and 2% on complex<float> across an A100 sweep, and 7% and 5% once the matrices fit in L2 and the copy is no longer bounded by main memory. Give the two backends separate strided tables rather than one compromise, as each is built against a single vendor's runtime. The unit-stride table is unchanged: the two machines agree on it to within 0.4% for every type. Every variant still fits the 17 KB budget, which on NVIDIA covers the 64 KB parts. Also record that geam was measured, not assumed, to be the slower option at unit stride on NVIDIA: on an A100 the kernels win from 1024x1024 up, by 2-231% pipelined and by up to 29% with a wait after every call. Co-authored-by: Cursor <cursoragent@cursor.com>
Select measured gfx90a and gfx942 tile geometries at runtime while retaining the gfx950 defaults for other AMD devices. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
omatcopy2for the cuBLAS and rocBLAS backends with portable asynchronous SYCL kernelsPerformance
complex<double>with non-unit strides: about 8.5% median improvement on two MI210 GPUsTest plan
wrong=0)git diff --checkThe standard omatcopy2 test binary currently aborts in the existing DPC++
ProgramManager::getDeviceKernelInfoassertion before executing the first test.Made with Cursor