Skip to content

[lapack][rocsolver] Use the 64-bit pivot API for getrf and getrs - #762

Open
zjin-lcf wants to merge 1 commit into
uxlfoundation:developfrom
zjin-lcf:feature/rocsolver-64bit-pivots
Open

[lapack][rocsolver] Use the 64-bit pivot API for getrf and getrs#762
zjin-lcf wants to merge 1 commit into
uxlfoundation:developfrom
zjin-lcf:feature/rocsolver-64bit-pivots

Conversation

@zjin-lcf

Copy link
Copy Markdown
Contributor

Summary

The rocSOLVER counterpart of #759, which does the same for cuSOLVER (#230).

oneMath types ipiv as int64_t, while the legacy rocSOLVER entry points take rocblas_int. getrf and getrs therefore each allocated a temporary 32-bit array and ran a cast kernel around the call, and in the USM getrs that temporary also forced a queue.wait() before it could be released.

rocSOLVER has accepted int64_t pivots since 3.26 (ROCm 6.2), so both routines now pass the user's ipiv straight through. That removes four temporary allocations, their cast kernels and the blocking wait. The 64-bit getrf reports info as int64_t rather than int, so get_rocsolver_devinfo and lapack_info_check are templated on the info type. ONEMATH_ROCSOLVER_HAS_64BIT_PIVOTS keeps the previous conversion path for rocSOLVER older than 3.26, so the minimum supported ROCm is unchanged.

Because the 64-bit entry points take the dimensions as int64_t, getrf and getrs no longer need the overflow_check that rejected sizes above the 32-bit limit.

Scope, and how this differs from #759

rocSOLVER has no sytrf_64, and getri and the getrf_batch, getrs_batch and getri_batch variants are unimplemented in this backend, so getrf and getrs are the only routines affected. There is nothing here matching #759's scratchpad-tail change, and sytrf keeps its existing conversion path.

As in #759, this reduces synchronization without making the USM paths fully asynchronous: getrf still blocks in lapack_info_check, which is pre-existing behaviour shared by the rest of the backend and left for separate work.

Upstream rocBLAS dependency above 2^28 rows

The API now accepts dimensions past 2^31, but actually factorizing a matrix with more than 2^28 rows also needs the rocBLAS fix from ROCm/rocm-libraries#10831, which I filed while testing this. rocblas_internal_scal_launcher_64 advances the alpha scalar pointer once per n-chunk, so getrf_64 either faults with hipErrorIllegalAddress or returns a silently wrong factorization for any column longer than 2^28.

This is not a regression introduced here: it affects every rocSOLVER release that has the 64-bit API (verified on 6.2.4, 6.3.4, 6.4.4, 7.0.1, 7.1.1, 7.2.0 and 7.2.4), the fix is one line in rocBLAS, and nothing at or below 2^28 rows is affected. Reviewers should just know that the "dimensions beyond 2^31" part of this change is not usable until that fix lands. I am happy to gate the 64-bit path on a fixed rocBLAS version, or to cap rows at 2^28 with the legacy path above it, if you would prefer this not to advertise a size range that upstream cannot yet deliver.

Test plan

Built against ROCm 7.2.4 with DPC++ and tested on an AMD Instinct MI300A (gfx942).

  • Functional LAPACK tests, both compile-time and run-time dispatch: 204 passed, 0 failed, 364 skipped, with identical results in both modes. Baseline develop on the same machine gives 180 passed, 0 failed, 364 skipped. The skips are the batch variants this backend does not implement.
  • The 24 newly passing tests are the Getrf and Getrs accuracy and dependency suites for all four types in buffer and USM form. To be precise about why they were not passing before: this DPC++ build trips a pre-existing assertion (It != m_DeviceKernelInfoMap.end()) on kernels compiled into a backend shared library, which the legacy path's cast kernels hit and the 64-bit path avoids by launching no kernels at all. On a toolchain without that problem both paths pass, so this is not a correctness change — the routines are simply now exercised locally.
  • Three Sytrf suites still abort from that same pre-existing assertion, unchanged from baseline; sytrf is untouched by this PR.
  • int64_t pivots verified end to end at m = 2^31 + 1024 (8 GiB, n = 1, largest entry at row m - 512): ipiv[0] = 2147484161, which is 514 past INT32_MAX and would have truncated to -2147483135 in the old 32-bit array. a[0] and the scaled column are correct. This run needs [rocBLAS] scal_64: alpha pointer advanced by the n-chunk offset corrupts n > 2^28 with stride_alpha != 0 (breaks rocsolver_?getrf_64) ROCm/rocm-libraries#10831; I validated it with that one-line fix applied at runtime through an LD_PRELOAD shim, cross-checked by confirming the shim also makes a standalone rocBLAS reproducer of the bug pass.
  • Version-guarded fallback compile-checked against real headers for rocSOLVER 3.24 (ROCm 6.0.3, legacy conversion path) and 3.26 through 3.32 (ROCm 6.2.4, 6.3.4, 6.4.4, 7.0.1, 7.1.1, 7.2.0, 7.2.4).
  • clang-format clean against the repository's _clang-format.

oneMath types ipiv as int64_t while the legacy rocSOLVER entry points take
rocblas_int, so getrf and getrs each allocated a temporary 32-bit array and ran
a cast kernel around the call. In the USM getrs that temporary also forced a
queue.wait() before it could be released.

rocSOLVER has accepted int64_t pivots since 3.26 (ROCm 6.2), so both routines
now pass the user's ipiv straight through, which removes the four temporary
allocations, their cast kernels and the blocking wait. The 64-bit getrf reports
info as int64_t rather than int, so the devinfo readback is templated on the
info type. Earlier rocSOLVER versions keep the conversion path, leaving the
minimum supported ROCm unchanged.

Because the 64-bit entry points take the dimensions as int64_t, getrf and getrs
no longer need the overflow_check that rejected sizes above the 32-bit limit.
Note that actually factorizing a matrix with more than 2^28 rows additionally
requires the rocBLAS fix from ROCm/rocm-libraries#10831: its 64-bit scal
launcher advances the alpha scalar pointer once per n-chunk, so getrf_64 either
faults or returns a silently wrong factorization for longer columns. With that
fix applied, an m = 2^31 + 1024 factorization returns the correct pivot index
2147484161, which is the case the legacy 32-bit array could not represent.

Co-authored-by: Cursor <cursoragent@cursor.com>
@zjin-lcf
zjin-lcf requested a review from a team as a code owner August 14, 2026 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant