Repository navigation
Conversation
Use the tightest visible cgroup CPU quota together with process affinity and host CPU count when sizing the ExecuTorch quantized GEMM compiler pool. Handle both cgroup versions and retain the existing 32-worker maximum on unrestricted hosts. CPU-only regression tests cover quota, affinity, namespace mount roots, parent limits and fallback behavior. Authored with OpenAI Codex.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24889
Note: Links to docs will display an error until the docs builds have been completed. ❌ 3 New Failures, 1 Cancelled Job, 4 Unrelated Failures, 1 Unclassified FailureAs of commit 78b04f7 with merge base 6150ca0 ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:
CANCELLED JOB - The following job was cancelled. Please retry:
BROKEN TRUNK - The following jobs failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
Rename backends/cuda/_cpu_count.py to backends/cuda/utils/cpu_count.py and split cgroup parsing into small helpers so flake8 C901 passes. Behavior is unchanged.
SaturatedCpu started one spinner per host CPU. Under a cgroup quota that oversubscribes the quota now that the quantized GEMM compile pool is sized to the usable CPUs, which starved the pool and pushed the int4 under-load test past its job timeout. Spawn one spinner per usable CPU instead.
Every recent kquant-17gb dflash-image-batching run, on main and on pull requests, was cancelled at the 240 minute limit while export_dflash was still running (more than 3.5 hours into the export). Raise that matrix entry to 480 minutes and keep the two kquant-dynamic entries at 240.
test-muse-glimmer-cuda-e2e runs three A100 jobs, and the kquant-17gb dflash-image-batching export alone takes about five hours, so running it on every pull request push costs many A100 hours per iteration. Gate the job on pushes to main; pull requests, ciflow tags and manual dispatch no longer start it. Pull requests keep the unittest-cuda-* jobs, including unittest-cuda-muse-glimmer-batching. A CI script test pins the gate.
…hing e2e
Each forward_{N} is its own AOTI compile and autotune. On the 11-vCPU A100
runner the full 1..64 set does not fit 240 minutes (#25018 still timed out
with this alone, so the 480-minute timeout stays); 1,4,16,64 exercises the
same CUDA executor padding path with three fewer compiles.
Export forward_{1,4} and a single draft owner in CI, and drop the job's
480-minute timeout back to the shared 240. Even with the CPU-quota-aware
compile pool and forward_{1,4,16,64}, the export was still compiling the
draft methods at the 240-minute mark (#25018); forward_16/64 and
draft_forward_{2,3,4} account for about two of its hours. The run still
covers static and dynamic target steps, vision, and DFlash drafting on the
batching runtime.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Quantized GEMM compilation sizes its worker pool using os.cpu_count(), which
continues to report host CPUs when a container limits CPU time with a cgroup
quota. Size the pool from the minimum of host CPU count, process affinity and
the tightest visible CPU quota, retaining the existing maximum of 32 workers.
The CPU-only helper handles cgroup v1 and v2, parent limits, mount roots and
cgroup namespace roots. Missing or unreadable quota information falls back to
the existing CPU count and affinity behavior. Fractional CPU quotas round up to
a positive worker count. The helper and its tests are included in Buck rules.
CI triggering and the four-hour Muse E2E timeout remain unchanged for the
initial CI verification.
Validation
14 CPU-only regression tests pass. Black and git diff --check pass.
Runtime probes on the same host using the patched _compile_workers():
Full Muse CI validation is pending PR creation. These probes validate worker
sizing; they do not establish end-to-end export performance with the fix.
Authored with OpenAI Codex.