Skip to content

Commit 6411f1e

Browse files
SS-JIApytorchbot
authored andcommitted
[executorch][native-ggml] Size the scheduler hash for graph leafs too
## Ulterior Motive Running Gemma 4 31B through ExecuTorch Native's GGML engine on CUDA, to find why it decodes slower than llama.cpp on the same GGUF. ## Rationale **What**: The backend scheduler is created for `2 * graph_size` tensors. **Why**: `ggml_backend_sched` hashes every node and every leaf of a graph it splits, and asserts that its hash set holds them all. It was sized for `graph_size`, which caps nodes and leafs separately, so a graph whose nodes and leafs together outgrew it failed that assertion, though either count alone fit. ## Details - **Before**: `ggml_backend_sched_new(..., graph_size, ...)`; ggml's own source notes the hash "needs to be size*2 to account for leafs". - **After**: `2 * graph_size`, room for a full graph of `graph_size` nodes and `graph_size` leafs. `test_graph_with_as_many_leafs_as_nodes` adds 2,100 separate constants to an input: 2,100 nodes and as many leafs, within the default graph size of 4,096. Authored with Claude Code. Differential Revision: [D130721991](https://our.internmc.facebook.com/intern/diff/D130721991/) ghstack-source-id: 469829014 Pull-Request: #24847
1 parent aff7e4c commit 6411f1e

2 files changed

Lines changed: 15 additions & 1 deletion

File tree

‎backends/native/runtime/engine/ggml/GgmlExecutor.cpp‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -492,11 +492,13 @@ class GgmlExecutable final : public EngineExecutable {
492492
std::ranges::transform(backends_, raw.begin(), [](const auto& backend) {
493493
return backend.get();
494494
});
495+
// The scheduler hashes a graph's leafs as well as its nodes, and a graph
496+
// holds up to graph_size of each.
495497
ggml_backend_sched_ptr scheduler(ggml_backend_sched_new(
496498
raw.data(),
497499
nullptr,
498500
static_cast<int>(raw.size()),
499-
options_.graph_size,
501+
2 * options_.graph_size,
500502
/*parallel=*/false,
501503
/*op_offload=*/true));
502504
if (!scheduler) {

‎backends/native/test/runtime/ggml/test_ggml_ops.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1510,6 +1510,18 @@ def test_index_put_reads_transposed_values_without_copies(self):
15101510
)
15111511
self.assertEqual(ops["CONT"], 0)
15121512

1513+
def test_graph_with_as_many_leafs_as_nodes(self):
1514+
# 2,100 additions of separate constants fit the default graph size of
1515+
# 4,096 nodes, but the scheduler also hashes their 2,100 leafs.
1516+
constants = [torch.randn(4) for _ in range(2100)]
1517+
1518+
def forward(x):
1519+
for constant in constants:
1520+
x = x + constant
1521+
return x
1522+
1523+
self._check(Lambda(forward), (torch.randn(4),))
1524+
15131525
def test_cached_graphs_refresh_inputs_and_persistent_cache(self):
15141526
t = Dim("t", min=1, max=8)
15151527
ep = torch.export.export(

0 commit comments

Comments
 (0)