Sync with Microsoft ONNX Runtime - 23082026 - #1267
Open
ai-fw-intg wants to merge 35 commits into
Open
Conversation
…osoft#32053) ### Description Add a new EP allowlist that includes WebGPU (`cpu_acl_cuda_dml_webgpu_eps`), and use it for the Level-2 `GeluFusion` and `BiasGeluFusion` transformers. This allows the erf-based GELU pattern (`Div` -> `Erf` -> `Add` -> `Mul` -> `Mul`, optionally preceded by a bias Add) to fuse into the WebGPU EP's kMSDomain Gelu and BiasGelu kernels instead of running as separate elementwise dispatches. This covers models using older opsets that the opset >= 20 Level-1 `GeluFusion` transformer does not handle. ### Performance Impact In native WebGPU build, the vision submodel of zero-shot image classification achieved a 1.11x wall-clock speedup on Panther Lake and a 1.15x speedup on Wildcat Lake. | Platform | Latency reduction | Speedup | |--------------------------|-------------------|---------| | Intel Wildcat Lake (WCL) | −13.0% | 1.15× | | Intel Panther Lake (PTL) | −9.9% | 1.11× | This PR addresses the `Gelu` and `BiasGelu` items listed in [microsoft#29841](microsoft#29841). --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Upgrade cutlass from 4.4.2 to 4.7 Upgrade cudnn-frontend from 1.24 to 1.27
### Description <!-- Describe your changes. --> - Use commit timestamps for plugin EP dev versions - Update set_plugin_ep_build_variables.py to use argparse ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Python dev package versions were using only the commit date. This prevented packages from multiple commits on the same day from being published. Using a finer granularity (from day to seconds) should mitigate it. It's probably good enough if we only expect to publish from `main` and release branches. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
To avoid OOM in cuda 13 plugin ep build: ``` ##[error]The agent worker exited with code 137, which means it ran out of memory. Make sure the agent (container) host has sufficient memory configured. ```
…t#32057) Widen each 8-lane int8 product group into the int32 accumulators before multiplying the second half of the packed K block. This avoids the signed int16 overflow that occurs when two `-128 * -128` products share a halfword lane. ### Description * Update the plain-NEON ARM64/AArch64 SymmQgemm S8 kernels to reduce each 8-lane product group into int32 before the next multiply. * Keep the ARM64 and AArch64 assembly implementations in sync. * Remove the temporary non-dotprod test guard introduced in microsoft#31606 so the existing signed-input regression coverage runs on the plain-NEON path again. ### Motivation and Context Fixes microsoft#31573. The previous `smull` + `smlal` sequence accumulated two int8 products in a signed int16 lane before widening. For the extreme case, `(-128 * -128) + (-128 * -128) = 32768`, which overflows int16. The revised sequence reduces each product group into the int32 accumulators before processing the second half of the packed K block. ### Validation The existing signed-input regression test is re-enabled for non-dotprod ARM64. The repository also contains `onnxruntime_mlas_benchmark` with `SYMMQGEMM/SignedActivation`; no representative non-dotprod Arm64 hardware was available for a trustworthy throughput comparison, so no performance numbers are claimed here.
This pull request addresses edge cases in the `TensorScatter` operator, particularly around handling large or potentially overflowing `write_indices` values. It also improves test coverage for these scenarios, ensuring robust and correct behavior in both linear and circular modes. ### Bug fixes and safety improvements * Fixed a potential overflow bug in linear mode by updating the bounds check to prevent `write_indices` values that could cause overflow when added to `sequence_length`. (`onnxruntime/core/providers/cpu/llm/tensorscatter.cc`) * Refined the circular mode logic to correctly handle very large `write_indices` values, ensuring correct wraparound behavior without overflow. (`onnxruntime/core/providers/cpu/llm/tensorscatter.cc`) ### Test coverage enhancements * Added a test for linear mode that verifies the operator fails gracefully when `write_indices` addition would overflow, ensuring the new bounds check is enforced. (`onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc`) * Added a test for circular mode to confirm that very large `write_indices` values wrap correctly without overflow, verifying correct wraparound logic. (`onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc`) * Included `<limits>` header to support the use of `std::numeric_limits<int64_t>::max()` in tests. (`onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc`)
This pull request improves the robustness and correctness of handling external data references in ONNX Runtime, particularly focusing on rejecting invalid or ambiguous "location" entries and ensuring in-memory reference tags are not treated as file paths. The changes also enhance path validation logic and add new unit tests to cover these scenarios. **Validation and error handling improvements:** - Added a check in `ExternalDataInfo::Create` to reject duplicate "location" entries in the external data info, ensuring only one "location" is allowed per tensor. [[1]](diffhunk://#diff-df6326d7af04c52e54c61249ff8b1980da9bac81192ee2e63c5b2280d7b91e05R32) [[2]](diffhunk://#diff-df6326d7af04c52e54c61249ff8b1980da9bac81192ee2e63c5b2280d7b91e05L42-R47) - Updated path validation logic in `ValidateExternalDataPathFromDir` and `ValidateExternalDataPath` to explicitly reject in-memory reference tags (used for internal memory mapping) as invalid file paths, alongside empty and absolute paths. [[1]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L410-R417) [[2]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L423-R437) [[3]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L438-R446) [[4]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L448-R457) [[5]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L469-R477) [[6]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L497-R509) **Functionality and code clarity:** - Refined `HasExternalDataInMemory` to correctly detect in-memory references even if there are multiple "location" entries, improving reliability. **Testing enhancements:** - Added new unit tests to verify that duplicate "location" entries are rejected and that in-memory reference tags are correctly handled and rejected by validation logic. [[1]](diffhunk://#diff-d75ec5db9cc4642f78b6ff568aff6d10398fc211b0fb7c862d3ec88738e3eda6R146-R188) [[2]](diffhunk://#diff-d75ec5db9cc4642f78b6ff568aff6d10398fc211b0fb7c862d3ec88738e3eda6R771-R782) These changes collectively strengthen the validation of external data references, prevent ambiguous or invalid configurations, and ensure that in-memory tags are never misinterpreted as file paths.
This pull request improves the robustness and correctness of the `scale_by_axis` method in the `Initializer` class and adds new tests to ensure proper handling of edge cases, especially with empty tensors and invalid scaler inputs. **Improvements to `scale_by_axis` implementation:** * Updated the calculation of `block_size` and `num_blocks` in the `scale_by_axis` method of the `Initializer` class to use more precise and consistent dimension handling, addressing potential bugs with axis indexing. (`onnxruntime/core/optimizer/initializer.cc`) **Testing enhancements:** * Added a new test case (`ScaleByAxisEmptyTensor`) to cover scenarios where the target or scaler tensors are empty, ensuring that the method does not throw unexpectedly and returns the correct size. The test also checks for proper exception handling with invalid scaler sizes and data types. (`onnxruntime/test/optimizer/initializer_test.cc`) * Included the `span_utils.h` header in the test file to support the new test cases using spans. (`onnxruntime/test/optimizer/initializer_test.cc`)
This pull request updates the logic for determining whether a node's input is a valid initializer for fusion and adds a new unit test to ensure that initializers which can be overridden (i.e., initializers that are also graph inputs) are properly skipped during fusion. The main changes are: ### Logic Update * In `IsNodeValidForFusion`, the check for a valid initializer now uses `graph_utils::GetConstantInitializer`, ensuring that only constant initializers (not those that can be overridden by graph inputs) are considered valid for fusion. ### Testing Improvements * Added a new test, `FuseInitializersSkipsOverridableInitializer`, to verify that the transformer skips fusion for initializers that are also graph inputs and can be overridden. This test ensures the fusion transformation does not incorrectly modify such initializers. * Included the `graph_transform_test_builder.h` header to support the new test.
This pull request improves input validation for the `Normalizer` operator by adding a check to reject scalar (rank-0) inputs and introducing a corresponding unit test to ensure this behavior. The main changes are as follows: **Input validation improvements:** * Added a check in `normalizer.cc` to return an error if the input tensor has rank 0, ensuring only rank 1 or 2 inputs are accepted. **Testing enhancements:** * Added a new test case in `normalizer_test.cc` (`ScalarInputRejected`) to verify that scalar inputs are correctly rejected with an appropriate error message.
This pull request strengthens input validation for the Conv operator by adding explicit checks for the bias tensor shape and size, and adds a new unit test to verify this behavior. The main changes are as follows: **Input validation improvements:** * Added a check in `Conv<T>::Compute` (and the float specialization) to ensure that the bias tensor `B`, if provided, is a 1D tensor whose size matches the number of output channels (`M`). If not, an informative error message is returned. [[1]](diffhunk://#diff-17bd0d956bdc88650a3b0fc11efe67ee921f0e2f0d5e5ab3e73925cd4bb510d7R75-R77) [[2]](diffhunk://#diff-17bd0d956bdc88650a3b0fc11efe67ee921f0e2f0d5e5ab3e73925cd4bb510d7R261-R263) **Testing enhancements:** * Introduced a new test case `Conv2D_InvalidBiasSize` in `conv_op_test.cc` to verify that the Conv operator correctly fails when the bias tensor does not have the expected size, ensuring the new validation logic works as intended.
This pull request improves the handling and validation of empty set reductions in the ONNX Runtime CPU reduction operators. It enforces stricter checks for the axes input, prevents unnecessary memory operations, and adds comprehensive tests to verify correct behavior for various edge cases. **Validation and Error Handling Improvements:** - Enforces that the `axes` tensor input must be a 1D vector and only processes it if it is present, improving robustness and error messages for invalid input shapes. - Adds a check to ensure that memory copying only occurs when the input tensor is non-empty, avoiding redundant operations. **Logic and Flow Adjustments:** - Refactors the order of reduction logic to first handle empty axes cases before checking for empty set input, ensuring correct execution flow and output. [[1]](diffhunk://#diff-61c3aeab1bffa9fc6cac3ace83df3a778d14c8f8d02a53ab46e5b776e567cdd4L971-R986) [[2]](diffhunk://#diff-61c3aeab1bffa9fc6cac3ace83df3a778d14c8f8d02a53ab46e5b776e567cdd4L1016-R1031) **Testing Enhancements:** - Adds new tests to verify: - Reduction when the optional `axes` input is missing (should reduce all dimensions). - Validation that the `axes` tensor must be a vector, with expected error messages for invalid cases. - Correct behavior for the `noop_with_empty_axes` attribute, both when axes are omitted and when an empty axes tensor is provided.
This pull request enhances the validation logic for segment embedding inputs in the quantized `QEmbedLayerNorm` operator and significantly improves test coverage to ensure robustness against partial or inconsistent segment embedding input scenarios. **Validation Logic Improvements:** - Enforces that all segment embedding-related inputs (`segment_ids`, `segment_embedding`, `segment_embedding_scale`, `segment_embedding_zero_point`) must be provided together or all omitted, preventing invalid partial configurations. [[1]](diffhunk://#diff-ba9db626fd260b1ca5d0794268c616200df2929f2cf66cf67c383634a3e66727R190-R191) [[2]](diffhunk://#diff-ba9db626fd260b1ca5d0794268c616200df2929f2cf66cf67c383634a3e66727R203-L202) **Test Suite Enhancements:** - Refactors the test harness (`RunTest`) to allow fine-grained control over which segment embedding inputs are present, using a bitmask for flexible test scenarios. - Updates test input logic to conditionally add each segment embedding input based on the bitmask, enabling the simulation of all possible partial input combinations. [[1]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL67-R72) [[2]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL82-R87) [[3]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL114-R119) [[4]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL140-R145) - Adds a new test, `PartialSegmentInputsRejected`, which systematically verifies that any partial provision of segment embedding inputs is correctly rejected by the operator, aligning with the new validation logic. - Modifies the test execution to expect failures with a clear error message when partial segment embedding inputs are provided.
This pull request improves input validation for the `ScatterND` operator and adds a corresponding unit test to ensure that invalid input is properly rejected. Input validation: * Added a check in `scatter_nd.h` to ensure that the last dimension of the `indices` tensor is at least 1, returning an error if it is not. Testing: * Added a unit test in `scatter_nd_op_test.cc` (`ScatterND_rejects_zero_index_depth`) to verify that the operator rejects `indices` tensors with a zero-sized last dimension.
This pull request updates the ONNX Runtime Python bindings to improve memory management and API flexibility for asynchronous inference sessions. The main changes ensure that Python objects used as inputs to `run_async` are properly referenced and not prematurely garbage collected, and that the API is more Pythonic by accepting `py::object` for session and run options. It also adds a test to verify that input arrays are kept alive during asynchronous execution. **Improvements to memory management and API flexibility:** * The `AsyncResource` struct now stores references to Python feed objects, the session, and run options to ensure they are kept alive for the duration of async execution (`onnxruntime/python/onnxruntime_pybind_state.cc`). * The `run_async` binding now accepts `py::object` for the session and run options, and internally casts them as needed. This makes the API more Pythonic and flexible (`onnxruntime/python/onnxruntime_pybind_state.cc`). * When preparing feeds for `run_async`, the code now stores each input object in `feed_objects` before creating the corresponding `OrtValue`, ensuring the Python objects are not garbage collected too early (`onnxruntime/python/onnxruntime_pybind_state.cc`). **Testing and validation:** * A test using `weakref` was added to verify that input arrays passed to `run_async` remain alive until the callback is invoked, preventing premature garbage collection (`onnxruntime/test/python/onnxruntime_test_python.py`) [[1]](diffhunk://#diff-bc2d3954a8ed883e3036dc7a675c7ba13b6299e1079728e7b6069cab1f49479cR690) [[2]](diffhunk://#diff-bc2d3954a8ed883e3036dc7a675c7ba13b6299e1079728e7b6069cab1f49479cR702) [[3]](diffhunk://#diff-bc2d3954a8ed883e3036dc7a675c7ba13b6299e1079728e7b6069cab1f49479cL712-R721). * The test also exercises explicit deletion and garbage collection of inputs, session, and run options to ensure robustness (`onnxruntime/test/python/onnxruntime_test_python.py`).
…OOB read (microsoft#29461) ### Description The Split operator's split-as-attribute path validates that each split size is non-negative (in the constructor), but the split-as-input-tensor path skips that check and only validates the aggregate (sum == axis dim, count == num_outputs). A crafted negative split size like [6, -2] on an axis of size 4 passes the aggregate check ( 6 + (-2) = 4 ) and causes the kernel to copy 6 rows from a 4-row input — an out-of-bounds read. Changes: • split.h ( PrepareForCompute ): Per-element >= 0 validation. Covers CPU, WebGPU, shared-provider paths. • cuda/tensor/split.cc ( PrepareForComputeLocal ): Same fix in the CUDA copy. • split_op_test.cc : NegativeSplitSizeInputTensor test with split = [6, -2] expecting failure. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d814472-441f-441d-bd46-931956efc1cd
### Description <!-- Describe your changes. --> Reject negative and out-of-range border and scale attributes without overflow. Align DirectML validation, handle empty CUDA outputs, and add provider-wide regression coverage. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fix Crop validation issues. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
### Description <!-- Describe your changes. --> Try to organize agent guidance so it is more extensible. E.g., add path-specific instructions, code review skill. Also moved skills to `.github/skills`. Added pointer to this location in AGENTS.md. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Goal is to improve agent code reviewing by providing guidance to follow. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…1957) **[MLAS] NCHWc HardSwish fusion for MobileNetV3 FP32 CPU inference** **Implementation of the hardswish fusion kernel for mobilenetv3 models.** **Implementation changes done:** - Fused the decomposed HardSwish pattern (Mul(x, HardSigmoid(x))) into the preceding NCHWc Conv as a native HardSwish activation, eliminating two standalone nodes per occurrence. - Added TryFuseNchwcHardSwish in nchwc_transformer.cc to detect and apply the fusion when the HardSigmoid parameters match HardSwish values (α=1/6, β=1/2) and the Mul consumes the same conv output. **Performance:** **Performance numbers on AMD Ryzen AI 9 365 (Strix Point)** NOTE: Performance taken on 31st July <img width="927" height="176" alt="image" src="https://github.com/user-attachments/assets/6246ceb6-fb2d-4d95-8768-dc808d1ee126" /> **STRIX 365 configration:** AMD Ryzen AI 9 365 (Strix Point) w/ Radeon 880M **Target Model:** MobilenetV3-Large and MobilenetV3-Small (FP32, CPU) **Unit Tests** **test_nchwc_hardswish.cpp** - Verifies TryFuseNchwcHardSwish correctly fuses the Mul(x, HardSigmoid(x)) diamond pattern into the preceding NCHWc Conv - Confirms fusion is not applied when α/β do not match HardSwish values (standalone HardSigmoid falls through to standard activation path) - Top-1 accuracy unchanged after fusion
## [WebGPU] PagedAttention: direct paged decode, fused paged prefill, Unpack/Repack skip (Phase 2 partial) This PR is the Phase 2 follow-up to microsoft#31611. It replaces the "always gather + always Unpack/Repack" v1 fallback with two paged-aware FlashAttention programs that read the paged KV cache directly, and a fast path that lets FA consume the packed varlen Q buffer without materializing padded BSNH scratch. Net effect: **~2× faster decode, ~1.15× faster uniform prefill, ~1.3× faster varlen prefill** on the shape matrix below, with no regressions. The Phase 1 gather-then-flash fallback shipped in microsoft#31611 remains intact and still runs on adapters / configs where the paged-aware shaders can't safely dispatch (see `Correctness invariants` below). ### What's shipped 1. **Direct paged split-reduce decode.** `FlashAttentionPagedDecodeQKV` + `FlashAttentionPagedDecodeVxReduce` index `key_cache` / `value_cache` directly through `block_table`. Selected when `max_seqlen_q < 32` — mirrors the dense-FA split-reduce threshold. Eliminates the dense K/V scratch and its gather bandwidth for every decode step. 2. **Fused paged prefill.** `FlashAttentionPagedPrefillProgram` is a straight port of the dense-FA prefill shader's shared-memory path with page-table-aware K/V tile loads (`bert/flash_attention_paged_prefill.wgsl.template`). Supports fp16, BSNH Q, packed varlen Q (`q_varlen` template variant), and variable-Q-length causal masking via `seqlen_k` + `seqlens_q`. No attention_bias / head_sink / TurboQuant. 3. **Unpack/Repack skip fast paths.** When direct paged attention runs, we can hand FA a rank-4 view over the raw packed Q buffer instead of allocating padded BSNH scratch: - **Uniform mode** (`B * max_seqlen_q == token_count`): view is `[B, max_seqlen_q, N, H]`. Covers decode, `B==1` prefill, and equal-length batched prefill (the common continuous-batching case). - **Varlen mode**: view is `[token_count, 1, N, H]` plus `cumulative_seqlens_q`; only the fused paged-prefill shader can index it (`q_varlen`). Skipping Unpack+Repack removes 2 dispatches (~300–500 µs of CPU dispatch cost per Run on D3D12) plus a `B * max_seqlen_q * hidden * 2 B` scratch allocation (tens of MB at long prefill). ### Dispatch-count reduction | Route (no rotary, non-packed) | microsoft#31611 (merged) | This PR (shm-path adapters) | |---|---|---| | **Decode** (`max_seqlen_q < 32`) | Scatter + Gather + UnpackQ + DecodeQKV + DecodeVxReduce + Repack = **6** | Scatter + PagedDecodeQKV + PagedDecodeVxReduce = **3** | | **Prefill** (`max_seqlen_q ≥ 32`) | Scatter + Gather + UnpackQ + FlashAttention + Repack = **5** | Scatter + FlashAttentionPagedPrefill = **2** | Decode's FA is 2 kernels (split-K: QKV + VxReduce); prefill's FA is 1 kernel (`FlashAttentionProgram`). On configs where `ShouldRunFusedPagedPrefill` rejects (fp32, `head_size > 256`, `block_size < max_k_step`), the prefill row falls back to the 5-dispatch microsoft#31611 cascade; decode's direct paged split-reduce path has no such gate. Neither route is adapter-gated — the paged shaders use no subgroup intrinsics and run on every WebGPU adapter that meets the fp16 / shm-budget / alignment predicates. ### Correctness invariants **Prefill selection** consults one shared predicate: ```cpp bool ShouldRunFusedPagedPrefill(context, is_fp16, max_seqlen_q, head_size, block_size); ``` It rejects (→ gather-then-flash fallback) when any of: - `!is_fp16` — only fp16 variant is compiled today. - `max_seqlen_q < 32` — decode uses the split-reduce programs instead. - `head_size` exceeds the workgroup shared-memory budget (fp16: `head_size > 256`). - `block_size < max_k_step` — the fused shader assumes one K/V tile lives in one paged block (one `block_table` lookup per tile). `paged_attention_helper` only enforces `block_size >= 16` power-of-two; e.g. `block_size=16` with fp16 `head_size<=128` (`max_k_step=32`) would splice into a physically-adjacent block that isn't the next entry in the table. The fused paged-prefill shader uses only workgroup shared memory (no subgroup intrinsics), so there is no adapter-class gate — subgroup adapters (Qualcomm / AMD / Intel with subgroups) take the paged shm kernel directly instead of falling back to gather + dense-FA-subgroup. Because the same predicate gates the "skip `RunGatherKV`", "skip `q_padded` scratch", and "select fused shader" decisions, the three cannot drift. **Decode selection** (`max_seqlen_q < 32`) is a pure shape check — no adapter, dtype, or block-size gate. The direct paged split-reduce kernels (`FlashAttentionPagedDecodeQKV` + `FlashAttentionPagedDecodeVxReduce`) are the sole decode path when the kernel dispatches at all (fp16 is enforced at kernel registration, so no fp32 fallback is possible). Unlike fused prefill, the decode kernels do one `block_table` lookup per K/V slot rather than per tile, so they have no `block_size` alignment requirement. **WGSL correctness gotcha handled** in the fused prefill shader. `cumulative_seqlens_q` is `array<i32>` but row indices are `u32`. Both `loadq` and `writeo` explicitly cast (`u32(cumulative_seqlens_q[b]) + q_idx`); without the cast, tint surfaces the type-resolution failure as an opaque `absl::…raw_hash_map<>::at` at runtime. ### Performance Machine: dev-box discrete WebGPU adapter (D3D12), 24-core host. Google Benchmark harness at `onnxruntime/test/onnx/microbenchmark/paged_attention.cc`, `--benchmark_min_time=0.3s`, wall-clock timing via `UseManualTime()`. Earlier revisions of this PR included an `ORT_WEBGPU_PAGED_ATTENTION_USE_FUSED` env-var kill switch used for A/B measurement against the microsoft#31611 cascade. That toggle has been removed (the direct/fused paths are selected internally by shape and config; the numbers below are the reason). The A/B was performed by temporarily broadening the toggle locally to also force `use_direct_paged_decode=false` and `skip_unpack_repack=false`, so fused=0 exercised the exact gather-then-flash cascade shipped in microsoft#31611. All numbers below are with that broadened toggle; the broadening was reverted before final push. Column meanings: **nH** = num query heads, **nKV** = num KV heads, **H** = head dim. Shape families: - MHA_H64 (nH=16, nKV=16, H=64), MHA_H128 (nH=16, nKV=16, H=128) - GQA_Qwen (nH=14, nKV=2, H=128), GQA_Llama (nH=32, nKV=4, H=128) #### Decode (16 shapes) | Shape (B/nH/nKV/H/past) | this PR (µs) | microsoft#31611 (µs) | Speedup | |---|---:|---:|---:| | 1/16/16/128/2048 | 669 | 3612 | **5.40×** | | 2/16/16/128/512 | 627 | 3248 | **5.18×** | | 1/16/16/64/2048 | 861 | 3835 | **4.45×** | | 2/16/16/64/2048 | 907 | 3617 | **3.99×** | | 2/16/16/64/512 | 603 | 1369 | 2.27× | | 2/16/16/128/2048 | 2129 | 4816 | 2.26× | | 1/16/16/128/512 | 607 | 1150 | 1.89× | | 2/32/4/128/2048 | 1590 | 3010 | 1.89× | | 1/14/2/128/2048 | 979 | 1601 | 1.64× | | 2/32/4/128/512 | 656 | 990 | 1.51× | | 1/16/16/64/512 | 576 | 843 | 1.46× | | 2/14/2/128/512 | 694 | 984 | 1.42× | | 1/14/2/128/512 | 600 | 757 | 1.26× | | 2/14/2/128/2048 | 970 | 1194 | 1.23× | | 1/32/4/128/512 | 643 | 751 | 1.17× | | 1/32/4/128/2048 | 1434 | 1437 | 1.00× | Range **1.00×–5.40×**, geomean ~2.0×. Biggest wins on long-past MHA (H=128, past=2048) where gather bandwidth dominated. The one 1.00× row is a small-K/V-cache GQA case where gather cost was already low. #### Uniform prefill (24 shapes) Range **1.01×–1.25×**, geomean ~1.13×. Highlights (all wins): | Shape (B/nH/nKV/H/T) | this PR (µs) | microsoft#31611 (µs) | Speedup | |---|---:|---:|---:| | 1/32/4/128/128 | 1225 | 1530 | **1.25×** | | 2/14/2/128/128 | 1111 | 1385 | **1.25×** | | 2/16/16/64/128 | 766 | 948 | 1.24× | | 2/32/4/128/512 | 9723 | 12054 | 1.24× | | 1/16/16/128/128 | 856 | 1051 | 1.23× | | 2/16/16/128/1024| 17296 | 21029 | 1.22× | | 1/14/2/128/128 | 718 | 877 | 1.22× | | 2/32/4/128/128 | 1640 | 1966 | 1.20× | | 1/16/16/64/128 | 750 | 891 | 1.19× | | 2/16/16/128/128 | 1294 | 1491 | 1.15× | *(14 more rows 1.01×–1.15×; full log in tree.)* Short-T shapes gain most from Unpack/Repack skip; long-T shapes are dominated by FA compute time. #### Varlen prefill (12 shapes, halving q_lens = `{max_T, max_T/2, max_T/4, …}`) Range **1.14×–1.73×**, geomean ~1.29×. | Shape (B/nH/nKV/H/maxT) | q_lens | this PR (µs) | microsoft#31611 (µs) | Speedup | |---|---|---:|---:|---:| | 4/16/16/128/512 | `{512,256,128,64}` | 5987 | 10375 | **1.73×** | | 4/14/2/128/512 | `{512,256,128,64}` | 5312 | 7815 | **1.47×** | | 4/32/4/128/512 | `{512,256,128,64}` | 11165 | 15427 | **1.38×** | | 4/16/16/128/1024 | `{1024,512,256,128}` | 20127 | 27661 | **1.37×** | | 2/32/4/128/512 | `{512,256}` | 8345 | 10732 | 1.29× | | 2/16/16/128/1024 | `{1024,512}` | 15121 | 19084 | 1.26× | | 4/14/2/128/1024 | `{1024,512,256,128}` | 17685 | 21562 | 1.22× | | 4/32/4/128/1024 | `{1024,512,256,128}` | 39571 | 47468 | 1.20× | | 2/14/2/128/1024 | `{1024,512}` | 13141 | 15430 | 1.17× | | 2/16/16/128/512 | `{512,256}` | 4528 | 5235 | 1.16× | | 2/32/4/128/1024 | `{1024,512}` | 28871 | 33383 | 1.16× | | 2/14/2/128/512 | `{512,256}` | 4067 | 4645 | 1.14× | Wins grow with batch size — bigger B means more of the padded-BSNH round-trip gets eliminated (B=4/maxT=512 packs only 46.9% of `B·maxT` tokens; the padded scratch microsoft#31611 allocates is >2× bigger than the actual data). ### Tests - `onnxruntime/test/contrib_ops/paged_attention_op_test.cc` `PagedAttention.EndToEnd_*` — 12/12 non-CUDA tests pass. Covers MHA, GQA, single/multi-batch, variable past lengths, empty tokens, packed QKV, rotary, mixed prefill+decode, cache aliasing via IO-binding. New: `EndToEnd_Prefill_MultiBatch_Varlen_Fused` (B=2, token_count=48 with q_lens (32,16), head_size=128, MHA) — regression test for the fused varlen prefill path. - Micro-benchmark harness `onnxruntime/test/onnx/microbenchmark/paged_attention.cc` — 52 registered shapes (16 decode + 24 uniform prefill + 12 varlen prefill). ### Related - Phase 1 (v1 fallback): microsoft#31611 (merged) - Schema extensions: microsoft#29912 (merged) - Design doc: `docs/design/webgpu_paged_attention.md` (updated in this PR) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
…ileClip-S0 model (microsoft#31958) **[MLAS] AVX-512 optimizations for MobileClip-S0 FP32 CPU inference** * Added a **16-wide AVX-512 Erf kernel** for standalone ONNX Erf operations, replacing the previous 8-wide implementation on AVX-512 hardware. * Implemented a **single-pass 16×16 AVX-512 NCHWc reorder transpose**, replacing the SSE2-based 4-wide sub-transpose approach for block-16 data reordering. The performance numbers taken in STRIX 365 with different thread configurations: <img width="974" height="116" alt="image" src="https://github.com/user-attachments/assets/a330b407-9d61-45f7-b6a5-9a08a774d6fc" /> NOTE: The performance numbers were tested on July 31st **STRIX 365 configuration:** AMD Ryzen AI 9 365 (Strix Point) w/ Radeon 880M **Target Model:** MobileClip-S0 (FP32, CPU) **Unit tests were added:** **test_erf.cpp** - MlasComputeErf vs std::erf within polynomial accuracy tolerance — sweeps buffer lengths straddling the 16-lane boundary to cover both the AVX-512 main loop and masked-tail path - Direct comparison of MlasErfKernelAvx512F vs the base MlasErfKernelFma3 with ≤1 ULP agreement — covers NaN propagation, ±inf, denormals, saturation - Measured divergence on AVX-512 hardware: 0 ULP (bit-exact); 1 ULP is kept as the cross-microarchitecture contract **test_reorder_input.cpp** - MlasReorderInputNchw (NCHW→NCHWc) vs scalar reference via memcmp - Sweeps channel counts 1–47: exact 16-channel blocks exercise the new MlasReorderInputNchwBlock16Avx512F fast path; partial blocks exercise the scalar tail - Multiple spatial sizes covered --------- Co-authored-by: Manogna-Sree <elisetti.manognasree@multicorewareinc.com>
### Description <!-- Describe your changes. --> ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
## Summary This PR pins GitHub Actions to full-length commit SHAs for improved security and reproducibility and adds a 7 day cooldown to Dependabot configuration for GitHub Actions. This work is described in more detail at https://aka.ms/action-pinning. ## Why? Pinning actions to commit SHAs prevents supply-chain attacks where a tag could be moved to point to malicious code. This is a recommended security best practice per the [GitHub Actions security hardening guide](https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#using-third-party-actions). This change mitigates the risk of tag retargeting to malicious code as seen in incidents like the [tj-actions/changed-files action compromise](https://www.stepsecurity.io/blog/harden-runner-detection-tj-actions-changed-files-action-is-compromised) or [codfish/semantic-release-action compromise](https://www.stepsecurity.io/blog/supply-chain-compromise-codfish-semantic-release-action) and improves the integrity and reproducibility of the CI/CD pipeline. ## What changed? **Action pinning:** Third-party action references in `.github/workflows/` that used mutable tag-based references (e.g., `actions/checkout@v4`) have been updated to full-length commit SHAs with a version comment (e.g., `actions/checkout@<sha> # v4`) using the [pinact](https://github.com/suzuki-shunsuke/pinact) tool. References that were already pinned to a SHA, or that used immutable release tags, were left unchanged. **Dependabot configuration:** `.github/dependabot.yml` has been updated to ensure a `github-actions` package-ecosystem section is present with a `cooldown` configuration (`default-days: 7`). If the file did not exist, it was created. If a `github-actions` section already existed, only the `cooldown` block was added or its `default-days` value was increased to 7 if it was lower. The 7-day cooldown provides a window for the community to detect and report compromised releases before they are automatically proposed as updates, reducing exposure to supply-chain attacks via newly published malicious versions. ## Is this safe to merge? Yes. The pinned SHAs correspond to the same commits that the existing tags pointed to. No behavioral changes in action execution are introduced. You can verify the pinned SHA value using the GitHub REST API (e.g., the commit hash for `actions/checkout@v7` can be found in the `sha` property in the JSON response for `GET https://api.github.com/repos/actions/checkout/commits/v7`). ## Additional Information For more information, please see https://aka.ms/action-pinning
…g it bakes into WGSL (microsoft#32048) ### Description `MatMulNaiveProgram` bakes three things into its generated WGSL that its pipeline cache hint did not declare. Any of them can serve a shader compiled for one configuration to a different one, which produces wrong results with no error and no warning. **1. The activation kind.** Two convolutions with identical shapes and different activations hashed to the same key, so the first shader compiled was reused for the second. **2. Activation parameters, via a formatting mismatch.** `Activation::ToString()` streamed floats through an unconfigured `std::stringstream` (6 *significant digits*) while `GetActivationSnippet()` emits them with `std::to_string` (6 *decimal places*). Values agreeing to 6 significant digits but differing as floats therefore produced one key and two different shaders. The maximally separated colliding pair is `1000015.0` / `1000025.0`, 160 float32 ULPs apart, both keyed as `1.00002e+06`. Fixed centrally in `ToString()` with `std::setprecision(std::numeric_limits<float>::max_digits10)`, so all six call sites that use it as a hint are corrected at once. **3. `is_channels_last`.** It selects between `bias[col]` and `bias[row + i]` in the same program, and `conv.cc` varies it, but it was absent from the hint. Added at both `MatMulNaiveProgram` call sites. Defects 1 and 3 are the same omission: five of the six programs on `main` that bake an activation into WGSL already declared it in their key, and four of those five also declared `is_channels_last`. `MatMulNaiveProgram` was the sole holdout on both counts. Defect 2 is pre-existing and shared (`main`'s `ToString()` is byte-identical) which is why the fix is in `ToString()` rather than at the call site. ### Motivation and Context Reachable from Conv today: a 1x1 kernel with unit stride and no padding lowers to a matmul, and when N and K are both under 8 it dispatches `MatMulNaiveProgram`. The regression tests put both convolutions in a **single graph** on purpose. Sequentially created sessions each get a fresh pipeline cache ( `WebGpuContextFactory` refcounts its contexts and destroys one at refcount zero) so running one activation per session cannot reproduce the bug. Concurrent sessions are different: contexts are held in a process-global map keyed by `context_id`, so two live sessions share one cache. That is what makes defect 3 reachable, and the layout test builds exactly that configuration. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…plates (microsoft#32115) ### Description Regenerates the WGSL in-tree golden fixtures. No production code changes; every file touched lives under onnxruntime/test/in_tree_golden/. Generated with: UPDATE_WGSL_GOLDEN=1 python tools/python/wgsl_template/test/run_tests.py Before: Ran 87 tests, FAILED (failures=2) After: Ran 87 tests, OK Three fixtures are new or rewritten: subgroup_matrix_gemm_8x16x16, subgroup_matrix_matmul_8x16x16, subgroup_matrix_matmul_pad_b. The remaining churn in im2col_matmul.h, oihw_to_ohwi.h, pad.h and string_table.h is index renumbering: the generator emits one globally numbered string table, so adding literals shifts every downstream __str_N index. Those files are unchanged once the indices are normalized. ### Motivation and Context The in-tree golden test compares the whole generated tree against committed fixtures, so it fails on main today. The subgroup-matrix templates were added or rewritten without regenerating the goldens. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
### Description Fused NEON kernel for LinearAttention ### Motivation and Context 3x speedup compared to generic dispatch
…de (microsoft#31146) Add S8S8 and U8S8 QGEMM compute kernels using the SVE i8mm svmmla instructions, selected at runtime whenever the processor supports SVE with the I8MM extension (HasArmSVE_I8MM). The kernels consume the exact packed A/B panels of the existing NEON smmla/ummla kernels (byte-identical packing code, PackedK=8, same RowSum/ColumnSum zero-point-correction layout), so results are bit-identical to the NEON path; only the inner compute differs (12x8 M-tiles with an 8x12 second shape, svmmla_s32 / svmmla_u32). The compute kernels ship as portable machine code in the style of Arm's KleidiAI library: every instruction is a raw word (GAS ".inst" / armasm64 "DCD" via aarch64/kai_asm_macros.h, macro set adopted verbatim from KleidiAI), so one generated file assembles under both the GNU assembler and Microsoft armasm64 with no SVE toolchain support required. The SVE intrinsics reference implementation (sve/qgemm_mmla_sve_impl.cpp) remains the regeneration source: sve/gen_sve_asm.py freezes it, verifying the code is fully self-contained (no relocations, no adrp/bl, no literal pools; compiled -fno-stack-protector) and the emitted words are byte-identical to the compiler's object code. The driver/pack/dispatch translation units use no SVE intrinsics and compile with plain AArch64 flags on any compiler, so the dispatch is not OS-gated; Windows runtime detection uses the PF_ARM_SVE_INSTRUCTIONS_AVAILABLE / PF_ARM_SVE_I8MM_INSTRUCTIONS_AVAILABLE feature constants (SDK-#ifdef-guarded), and the sources are wired through the existing cl /P + armasm64 pipeline. cmake option onnxruntime_SVE_QGEMM_ASM (default ON) selects the frozen machine code; OFF builds the intrinsics reference instead. M == 1 (GEMV-shaped) operations delegate to the NEON mmla dispatch on Linux, where those kernels exist: a single packed row cannot amortize the 2-row mmla pair structure, and the packed-B interchangeability makes the delegation valid for both the unpacked and prepacked paths (measured: M=1 prepacked cells go from up to 1.07x slower to parity). Measured on Cortex-X925/A725 (SVE VL=128, 5 big + 5 mid cores), full MLAS QGEMM benchmark (76 median cells, all four SignedA/UnsignedA x PackB/ NoPackB variants), pinned to the big cluster, versus the NEON baseline built from the same base commit (real-time medians, lower is better): Threads:4 (fits the big cluster) 0.84-0.86x (~1.17x speedup) Threads:16 (oversubscribed on 5c) 0.93-0.97x Threads:1, M=1 rows 0.98-1.02x (parity, after the M==1 delegation) All 76 cells 0.932x The frozen machine code measures at parity with the intrinsics build overall (0.932x vs 0.931x); the Threads:4 band pays ~1% for the kernel now being an out-of-line call rather than force-inlined into the operation loop. Correctness: full onnxruntime_mlas_test with the SVE dispatches active by default: 33213 passed, 0 failed. Results are bit-identical to the NEON kernels (exact integer accumulation over identical packed operands).
## Description Replace the obsolete `doxygen.nl` download URL for the pinned Doxygen 1.9.8 Linux binary with the official asset from the Doxygen GitHub release tag `Release_1_9_8`. The C API docs workflow has persistently failed while downloading Doxygen (runs [31917039689](https://github.com/microsoft/onnxruntime/actions/runs/31917039689), [31975339915](https://github.com/microsoft/onnxruntime/actions/runs/31975339915), and [32050624667](https://github.com/microsoft/onnxruntime/actions/runs/32050624667)). Because no fresh `onnxruntime-c-apidocs` artifact was produced before the previous artifact expired, the downstream ONNX Publish site run [32533178465](https://github.com/microsoft/onnxruntime/actions/runs/32533178465) failed at `Download C apidocs artifact`, blocking GitHub Pages deployment. This is intentionally separate from accessibility PR microsoft#32180. ## Validation - Confirmed the official release URL follows one redirect and returns HTTP 200. - Confirmed GitHub release tag `Release_1_9_8` contains `doxygen-1.9.8.linux.bin.tar.gz` (50,500,806 bytes). - Downloaded the archive successfully and verified gzip/tar integrity. - Confirmed extraction produces `doxygen-1.9.8/bin/doxygen` while preserving the workflow's existing extraction and invocation paths. - Parsed the workflow as valid YAML. - Dispatched the [C/C++ API docs workflow from this branch](https://github.com/microsoft/onnxruntime/actions/runs/32534100983); the install, Doxygen generation, and site staging steps succeeded, and the log reports `Doxygen version used: 1.9.8`. Artifact upload was skipped as expected because the ref is not `main`. ## Recovery after merge 1. Dispatch the **Update C/C++ API Docs** workflow on `main`. 2. Verify it uploads a fresh `onnxruntime-c-apidocs` artifact. 3. Rerun the failed GitHub Pages jobs from ONNX Publish site run 32533178465. No Doxygen version or unrelated workflow behavior is changed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…osoft#29820) ### Description - Update bundled Dawn to tag [`v20260817.213942`](https://github.com/google/dawn/releases/tag/v20260817.213942) (`0b66dc1aa0102b0147962820d372bd2eb3a329b4`). The tag contains the worker-pool API commit [`c3cf2a7`](google/dawn@c3cf2a7) plus 76 subsequent Dawn commits. - Add an ORT-specific `dawn::platform::Platform` for the default native bundled-Dawn instance. Its `CreateWorkerTaskPool()` override uses Dawn's default implementation with `max(2, std::thread::hardware_concurrency() / 2)` workers. - Keep the platform alive for the process lifetime because Dawn retains a non-owning pointer. - Link `dawn_platform` for both static and shared bundled-Dawn builds because the custom platform and public worker-pool factory are implemented by that library. - WebAssembly builds, external Dawn builds, and user-provided WebGPU instances keep their existing behavior. ### Motivation and Context Dawn's default worker task pool is limited to two threads. Models with many independent WebGPU pipelines can therefore bottleneck during asynchronous pipeline creation. Scaling the existing Dawn pool to half of the host's logical processors allows more pipeline compilations to proceed in parallel and reduces WebGPU initialization and first-run latency without consuming every CPU thread. The minimum of two also preserves Dawn's current capacity when `hardware_concurrency()` is small or returns zero. The required Dawn API landed upstream in [Allow configuring the default worker task pool](https://dawn-review.googlesource.com/c/dawn/+/330555). This PR consumes it from the subsequent tagged Dawn snapshot; no ORT-local Dawn patch is needed. ### Testing - Built the `onnxruntime` target on Windows RelWithDebInfo with `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON`, bundled Dawn, and the WebGPU EP enabled. - The configuration links the ORT custom platform against `dawn_platform` in both static- and shared-Dawn branches. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
August 22, 2026 20:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.