feat(rccl): enable multi-node MADEngine CI tests [AICOMRCCL-1332] by prasanna-amd · Pull Request #10396 · ROCm/rocm-systems

prasanna-amd · 2026-08-19T15:05:34Z

Summary

Adds MADEngine workload integration into RCCL CI, validated end-to-end on 2N/16GPU (MI355X, Ruby cluster)
Fixes multi-node blockers: SLURM node-pinning, QoS rejection, NCCL_SOCKET_IFNAME, Docker --network=none
Resolves rdma-core ABI mismatch (container rdma-core 50 vs host rdma-core 61) by bind-mounting the host's libibverbs and IB provider directory at the compiled-in search path
Fixes perf_entry_super.json metric name parsing so results report correctly (tok/s/GPU, TFLOPS/GPU)

JIRA ID : AICOMRCCL-1332

Validation (Ruby cluster, job 26928)

Llama 3.1 70B BF16 pretrain (2N/16GPU): 1,719 tok/s/GPU, 773 TFLOPS/GPU
Llama 3.1 70B FP8 pretrain (2N/16GPU): 2,848 tok/s/GPU, 1,280 TFLOPS/GPU
Bare-metal rccl-tests 2N all_reduce_perf: 105.3 GB/s avg bus bandwidth, 0 errors

Key changes

test_madengine.py: multi-node manifest generation, host IB lib bind-mounts, metric parsing fixes
therock-rccl-test-madengine.yml: new workflow for MADEngine workloads
rccl_ci_utils.py: shared CI utilities (GitHub summary, outputs)
test_jax_collective.py, test_pytorch_c10d.py: additional CI test scripts

Test plan

2N MADEngine Llama 3.1 70B training passes on Ruby (BF16 + FP8)
Bare-metal 2N rccl-tests all_reduce_perf passes
perf_entry_super.json parsing produces correct metrics
python3 -m py_compile passes on all new scripts

🤖 Generated with Claude Code

therock-pr-bot · 2026-08-19T15:06:14Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-08-19T15:06:17Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot

Pull request overview

This PR expands RCCL CI coverage by adding MADEngine workload execution (including multi-node) to the existing TheRock-driven CI pipeline, alongside shared helper utilities and test runners.

Changes:

Adds a new MADEngine test workflow and wires it into the scheduled RCCL CI on gfx950 (Ruby SLURM runner).
Introduces a MADEngine workload runner (test_madengine.py) that builds an RCCL overlay image, generates a manifest, runs MADEngine, and parses/records performance metrics.
Adds/duplicates JAX and PyTorch distributed smoke test runners and shared CI utilities under projects/rccl/ci/scripts.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
projects/rccl/ci/scripts/test_pytorch_c10d.py	PyTorch c10d NCCL test runner for CI-built RCCL with cloning + JUnit parsing + reporting.
projects/rccl/ci/scripts/test_madengine.py	MADEngine workload runner: overlay image build, manifest generation, execution, metrics parsing, regression checks, reporting.
projects/rccl/ci/scripts/test_jax_collective.py	JAX collective smoke test runner for CI-built RCCL with JUnit parsing + reporting.
projects/rccl/ci/scripts/rccl_ci_utils.py	Shared helpers for artifact discovery, JUnit parsing, GitHub summary/output, and notifications.
.github/workflows/therock-rccl-test-madengine.yml	New reusable workflow to run MADEngine workloads on Ruby SLURM runner and upload artifacts.
.github/workflows/therock-rccl-ci.yml	Changes scheduled cadence to nightly; adds `madengine_nodes` input and passes it through.
.github/workflows/therock-rccl-ci-linux.yml	Adds `madengine_nodes` input and a scheduled/dispatch MADEngine test job for gfx950.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

mkuznet1

LGTM

mkuznet1 · 2026-08-19T16:52:22Z

e4af659 → c3a9345 → 4694545 → 0af47b2
is what we also fixed in sglang-disagg: fix queue-pair creation on Broadcom Thor2 (rdma-core 63.0), and make every overlay s…

mkuznet1 · 2026-08-19T16:52:49Z

#163 (merged in madengine - fixes for ruby) - You can use this to reduce the overall delta (by reducing the number of patches in MADEngine that are currently in [feat(rccl): enable multi-node MADEngine CI tests AICOMRCCL-1332 by prasanna-amd · Pull Request #1…](#10396))

i-kosarev

My main and pretty much only concern is what was commented in #9055 about build-time gate that confirms the copied .so's hash/symbol matches the CI-built artifact: #9055 (comment)

prasanna-amd · 2026-08-20T17:18:28Z

@i-kosarev This is now addressed in commit aa2cdf8. The script extracts an md5 + version string (2.30.4-HEAD:e711c9e) from the CI-built librccl.so pre-flight, then after the run completes it parses the SLURM logs for RCCL version : and compares against the expected fingerprint. If they don't match, the run is failed.

Verified in job 27169 — full output:

INFO: RCCL fingerprint: md5=63ab3428628521278c77c034a23fe4f6 version=2.30.4-HEAD:e711c9e size=4100992
INFO: RCCL verification: RCCL verified: version=2.30.4-HEAD:e711c9e, artifact_md5=63ab3428628521278c77c034a23fe4f6, librccl_path=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib/librccl.so.1

Could you re-review when you get a chance?

i-kosarev · 2026-08-20T18:19:36Z

@i-kosarev This is now addressed in commit aa2cdf8. The script extracts an md5 + version string (2.30.4-HEAD:e711c9e) from the CI-built librccl.so pre-flight, then after the run completes it parses the SLURM logs for RCCL version : and compares against the expected fingerprint. If they don't match, the run is failed.

Verified in job 27169 — full output:
INFO: RCCL fingerprint: md5=63ab3428628521278c77c034a23fe4f6 version=2.30.4-HEAD:e711c9e size=4100992
INFO: RCCL verification: RCCL verified: version=2.30.4-HEAD:e711c9e, artifact_md5=63ab3428628521278c77c034a23fe4f6, librccl_path=/opt/venv/lib/python3.12/site-packages/_rocm_sdk_libraries/lib/librccl.so.1
Could you re-review when you get a chance?

Thanks — the fingerprint mechanism itself looks right, and job 27169 shows it
working. But as wired up, it doesn't run in CI.
The call is gated on --skip-overlay-build:

if args.skip_overlay_build and rccl_fingerprint.get("md5"):
  rccl_ok, rccl_msg = verify_rccl_replacement(...)

and therock-rccl-test-madengine.yml never passes that flag — the scheduled run
only gets --artifact-dir/--workload/--cluster/--nodes/--results-dir/--work-dir/--registry.
So the nightly path builds the overlay and verify_rccl_replacement() is never
called. The output you pasted (librccl_path=..., artifact_md5=...) is the
bind-mount path, i.e. the one branch that does verify.
Two more gaps in the check itself:

It reads log_files[0] from glob("*node_0.out"), so on a 2N run node_1 is
never verified — a library that reached only one node still passes. That's the
multi-node failure this job exists to catch.
The failure it raises (exit_code = max(exit_code, 1)) can be cleared later by
the "Overriding madengine exit code → 0" block if the parsed rows look complete.
To close this out: run the verification on the overlay path too (the expected
md5/version are known in both modes), assert it on every *node_*.out, and make
a verification failure non-overridable.


Its three asks, in one line each, in case you want to trim further:
1. Run the verification on the overlay path too — it's currently gated behind --skip-overlay-build, which CI never passes.
2. Check every *node_*.out, not just node_0 — otherwise a library that reached only one node passes.
3. Make a verification failure non-overridable — the exit-code override block can clear it.

prasanna-amd · 2026-08-20T20:11:49Z

@i-kosarev Addressed all three points in 60a1f51:

Runs in CI — removed the --skip-overlay-build gate; verification now fires in both overlay and bind-mount modes.
All nodes checked — iterates *node_*.out instead of just node_0; fails if any node has a missing or mismatched RCCL version.
Non-overridable — rccl_verification_failed flag prevents the exit-code override block from clearing a verification failure.

Tested against job 27169 (2N run): RCCL verified on 2 node(s): version=2.30.4-HEAD:e711c9e.

Could you re-review?

i-kosarev

I am good with the code, but we need actual GH run passing. Right now it has issues: https://github.com/ROCm/rocm-systems/actions/runs/32419399327

Remove --registry, docker login step, and packages:write permission. Images reach compute nodes via MAD_DOCKER_BUILDS — the registry push is unnecessary and the elevated permission causes startup_failure when called from therock-ci.yml which only grants contents:read.

The caller in therock-rccl-ci-linux.yml was still requesting packages:write for the madengine job, but the parent workflow only grants contents:read — causing startup_failure. Drop it from both the caller and callee since images reach nodes via MAD_DOCKER_BUILDS.

- Update TheRock checkout to a30607ad (2026-08-24) - Lift arch-specific env vars to workflow-level, replace case block with a simple arch guard - Split pip install into its own step for clearer error attribution

pip3 is not in PATH on ruby-linux-slurm-scale-runner nodes.

geomin12

please address concerns

overall lgtm. of course, this must be maintained by the RCCL team and must make sure madengine is available for CI to run and not block

geomin12

~~hm. i approved, but for some reason, this is back in my queue. had a few nits too~~ i see the update

amd-justchen · 2026-08-29T06:57:53Z

Will merge given code owner approval. The TheRock CI failures are only known infra issues, unrelated to the MADEngine CI-test change:

Linux MI455 Test / Test Sanity Check: Docker "all predefined address pools have been fully subnetted" (runner infra)
Test rocgdb-cpu (gfx94X): amd-smi static GPU sanity failure (exit 255)

[AICOMRCCL-1332] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add Llama-3.1-70B training workload (via MADEngine) to the RCCL CI pipeline on the Ruby cluster. The mechanism is a Docker overlay image that swaps the CI-built librccl.so into a rocm/primus base container, enabling end-to-end validation of RCCL builds against real training workloads without rebuilding the full PyTorch/ROCm stack. Key components: - test_madengine.py: orchestration script — overlay build, manifest generation, SLURM dispatch, live log metric extraction, JSONL perf datastore with per-precision rolling regression detection - therock-rccl-test-madengine.yml: workflow for MADEngine workloads - CI schedule changed from weekly to nightly (06:17 UTC) - CI scripts moved to projects/rccl/ci/scripts/ (from .github/scripts/) Clones pinned: madengine at ec4de0b58c49, MAD at 688828bd9d4a on the mad-rccl branch. HF token passed via process environment only (not written to manifest artifact). Validated single-node (8x MI325X): BF16: 773.9 TFLOP/s/GPU, 1722 tokens/s/GPU (50/50 iterations) FP8: 1249.4 TFLOP/s/GPU, 2780 tokens/s/GPU (50/50 iterations) [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

runs Fixes discovered during manual validation on Ruby cluster: - Add --network=none to Docker overlay build (bridge not available on all compute nodes) - Pin SLURM job to the overlay build node when no registry is configured (image only exists locally) - Disable madengine node health check that overrides nodelist - Strip multi-NIC config for single-node runs (Gloo requires all listed NICs to exist, unlike NCCL) - Fix MAD scripts path (primus_megatron-lm, not primus/megatron-lm) - Use full SHA for MAD pin - Smart MAD clone: verify HEAD before fetch+checkout - Handle PermissionError when saving run artifacts to shared dir Validated: 1N/8GPU llama-3.1-70b-training on Ruby BF16: 763.2 TFLOP/s/GPU, 1698.4 tok/s/GPU FP8: 1254.7 TFLOP/s/GPU, 2792.2 tok/s/GPU [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

=?UTF-8?q?=E2=80=94=20manifest,=20provenance,=20results?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix critical issues from mkuznet1 and i-kosarev review: Manifest structure (generate_manifest): - Move slurm config into deployment_config.slurm (was root-level, ignored by madengine) - Move env vars into context.docker_env_vars and deployment_config.env_vars (root-level env was dead) - Move container mounts into context.docker_mounts (root-level container_mounts was dead) - Add multiple_results to built_models card so madengine can produce per-precision structured output (root cause of empty metrics) - Set n_gpus to -1 (madengine resolves per-node) - Leave training_precision empty (card runs both BF16 and FP8) - Add GLOO_SOCKET_IFNAME to docker_env_vars - Remove MAD_MULTI_NODE_RUNNER (noise for primus launcher) - Add qos to deployment_config.slurm (was silently dropped) - Add docker_run_options from reference template Simplify run_madengine: - Remove redundant additional_context (slurm, distributed, env_vars) since manifest now has deployment_config in the right place and madengine merges it automatically RCCL commit provenance (get_rccl_commit): - Remove git rev-parse HEAD fallback — in CI this returns TheRock's pinned commit (constant), causing stale image cache hits - Use RCCL_COMMIT_HASH env, then GITHUB_RUN_ID, then sha256 of librccl.so for unique image tags Push tag format (build_rccl_overlay_image): - Replace both / and : in base image name to avoid invalid Docker reference with two colons Results parsing: - Replace parse_perf_csv with parse_perf_results reading perf_entry_super.json (31 fixed columns, per-precision rows) - Fall back to perf.csv with all rows (was collapsing to last row) - Use per-run status in datastore (was stamping one status on all) - Copy perf_entry_super files to run artifacts - Require metric_value for exit code override (was only checking iteration count) Workflow YAML: - Add secrets: inherit to therock-test-madengine job (HF_TOKEN was silently empty) - Add packages: write permission (needed for GHCR push) - Upload perf_entry_super files as CI artifacts [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

MADEngine metrics Drive regression checks and datastore writes from structured output (perf_entry_super.json) instead of live-log scraping. Addresses three review items from mkuznet1 (Aug 12): - Filter structured rows by metric_key and key by precision instead of overwriting metric_value on every row (last-row-wins bug) - Invert priority: structured results are primary for regression and datastore, live-log scraping is fallback only when structured data is missing - Build a single precision_results list consumed by all downstream stages (exit-code override, regression, datastore, report) Also fixes: - Preserve header-only runs (precision detected, zero iterations) as incomplete instead of silently discarding them - Record status=fail with metric_value=None when no results exist (was recording pass with no metric) [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

- Only pin SLURM job to build node for single-node runs; multi-node needs SLURM to allocate across nodes freely - Remove --network=none from docker build so overlay push to GHCR works - Drop vip_prio QoS (DenyQos on meta64 partition rejects it) [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

=?UTF-8?q?=5FSOCKET=5FIFNAME=20=E2=80=94=20compute=20nodes=20only=20have?= =?UTF-8?q?=20fenic0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Gloo fails with "Unable to find address for: enp49s0f0np0" because that interface doesn't exist on Ruby compute nodes. Only fenic0 is present across all nodes. [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

The container's libibverbs/bnxt_re provider doesn't match the host kernel's RDMA driver, causing ibv_create_qp to fail with "Bad address" on multi-node runs. Mount the host's /usr/lib64/libibverbs* into /host_ib_libs/ and set LD_LIBRARY_PATH + LIBIBVERBS_DRIVER_PATH so RCCL uses the host-matching verbs providers. Controlled by the existing mount_host_ib_libs flag in CLUSTER_CONFIGS. [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

docker_mounts convention is {container_path: host_path}, not the reverse. The previous commit had the mapping backwards, mounting empty container paths onto the host instead of host libs into the container. [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

mount Container ships rdma-core 50 (rdmav34 ABI) but Ruby hosts run rdma-core 61 (rdmav59). The ABI mismatch causes ibv_create_qp to fail with "Bad address" on multi-node runs. Mount the host's libibverbs.so, provider directory, and libibumad.so over the container's symlink targets so the existing symlinks resolve to the host's rdma-core 61 binaries. [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

The host's libibverbs.so has /usr/lib64/libibverbs/ hardcoded as its provider search path. Previous mounts placed providers at the container's Ubuntu path (/usr/lib/x86_64-linux-gnu/libibverbs/) which the host binary never searches, causing "cannot open shared object" for every rdmav59 provider. Mount the provider directory at /usr/lib64/libibverbs/ and replace the container's libibverbs.so.1 with the host's via the standard Ubuntu library path. Also use soname symlinks instead of version-specific filenames for portability. [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

correctly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit madengine emits metric names like "tok_per_s_per_gpu" and "TFLOPS_per_gpu" in perf_entry_super.json, but the parser compared against our config key "tokens_per_second_per_gpu" — every row was skipped, leaving precision_results empty and status=fail. Also: training_precision is empty in the JSON; the actual value lives in multi_results.precision. And madengine writes "SUCCESS" not "PASS". Fixes: - Map config metric_key to madengine aliases via _METRIC_ALIASES - Fall back to multi_results.precision when training_precision empty - Accept "SUCCESS" (case-insensitive) as a passing status - Capture TFLOPS from companion rows and attach to precision_results [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

When --skip-overlay-build is used, the container's bundled RCCL was used instead of the CI artifact. Now the artifact librccl.so (and kpack files) are bind-mounted over the container's SDK library paths so we actually test the intended RCCL build. [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

Extract version and md5 from the CI artifact pre-flight, then verify post-run that the container's RCCL version string matches. Catches cases where the container's bundled RCCL is used instead of the bind-mounted CI artifact. Works across all frameworks (JAX, Megatron, SGLang, vLLM) since RCCL always prints its version during init. [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

non-overridable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address review feedback: 1. Remove --skip-overlay-build gate so verification runs in both overlay and bind-mount modes (the nightly CI path builds an overlay). 2. Check every *node_*.out log, not just node_0 — a library that reached only one node still fails. 3. Track rccl_verification_failed separately so the exit-code override block cannot clear a verification failure. [rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

[rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

systems-assistant · 2026-08-31T14:19:52Z

TheRock Submodule Bump Activity

Newest first

2026-08-31 — Included in TheRock via ROCm/TheRock#7760.

prasanna-amd requested review from a team and a lite review from Copilot August 19, 2026 15:05

prasanna-amd requested a review from a team as a code owner August 19, 2026 15:05

github-actions Bot added github actions Pull requests that update GitHub Actions code project: rccl labels Aug 19, 2026

therock-pr-bot Bot added the Not ready to Review label Aug 19, 2026

Copilot started reviewing on behalf of prasanna-amd August 19, 2026 15:07 View session

systems-assistant Bot added the organization: ROCm label Aug 19, 2026

Copilot AI reviewed Aug 19, 2026

View reviewed changes

mkuznet1 approved these changes Aug 19, 2026

View reviewed changes

mkuznet1 reviewed Aug 19, 2026

View reviewed changes

Comment thread projects/rccl/ci/scripts/test_madengine.py Outdated

i-kosarev requested review from ddurnov and i-kosarev August 19, 2026 16:49

i-kosarev reviewed Aug 19, 2026

View reviewed changes

i-kosarev approved these changes Aug 20, 2026

View reviewed changes

ddurnov approved these changes Aug 21, 2026

View reviewed changes

prasanna-amd requested a review from geomin12 August 21, 2026 15:27

prasanna-amd force-pushed the users/prasanna-amd/rccl-ci-madengine-multinode branch from 9e03f79 to eba4ea4 Compare August 24, 2026 17:45

geomin12 reviewed Aug 24, 2026

View reviewed changes

prasanna-amd removed the Not ready to Review label Aug 24, 2026

therock-pr-bot Bot added the Not ready to Review label Aug 24, 2026

prasanna-amd requested a review from geomin12 August 24, 2026 23:02

Prasannakumar Murugesan added 4 commits August 25, 2026 23:00

fix: re-pin MAD to 63867e6a6e42

737b4d8

refactor: address workflow review feedback (hash, env vars, step split)

09f7072

- Update TheRock checkout to a30607ad (2026-08-24) - Lift arch-specific env vars to workflow-level, replace case block with a simple arch guard - Split pip install into its own step for clearer error attribution

prasanna-amd force-pushed the users/prasanna-amd/rccl-ci-madengine-multinode branch from 25c31d6 to 09f7072 Compare August 25, 2026 23:00

prasanna-amd requested a review from a team August 26, 2026 02:56

i-kosarev mentioned this pull request Aug 26, 2026

feat(rccl): integrate MADEngine workloads into CI [AICOMRCCL-1332] #9055

Closed

4 tasks

fix: use python3 -m pip on Ruby SLURM runners

dd6224d

pip3 is not in PATH on ruby-linux-slurm-scale-runner nodes.

geomin12 approved these changes Aug 27, 2026

View reviewed changes

Comment thread .github/workflows/therock-rccl-test-madengine.yml

Comment thread .github/workflows/therock-rccl-test-madengine.yml

geomin12 reviewed Aug 27, 2026

View reviewed changes

prasanna-amd removed the Not ready to Review label Aug 28, 2026

amd-justchen merged commit 2b51bf8 into develop Aug 29, 2026
86 of 93 checks passed

amd-justchen deleted the users/prasanna-amd/rccl-ci-madengine-multinode branch August 29, 2026 07:03

systems-assistant Bot pushed a commit to ROCm/rccl that referenced this pull request Aug 29, 2026

fix: re-pin MAD to 63867e6a6e42

ab6aee0

[rocm-systems] ROCm/rocm-systems#10396 (commit 2b51bf8)

Uh oh!

Conversation

prasanna-amd commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation (Ruby cluster, job 26928)

Key changes

Test plan

Uh oh!

therock-pr-bot Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Aug 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mkuznet1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mkuznet1 commented Aug 19, 2026

Uh oh!

mkuznet1 commented Aug 19, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

i-kosarev left a comment

Choose a reason for hiding this comment

Uh oh!

prasanna-amd commented Aug 20, 2026

Uh oh!

i-kosarev commented Aug 20, 2026

Uh oh!

prasanna-amd commented Aug 20, 2026

Uh oh!

i-kosarev left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geomin12 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

geomin12 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amd-justchen commented Aug 29, 2026

Uh oh!

Uh oh!

systems-assistant Bot commented Aug 31, 2026

TheRock Submodule Bump Activity

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

prasanna-amd commented Aug 19, 2026 •

edited

Loading

therock-pr-bot Bot commented Aug 19, 2026 •

edited

Loading

mkuznet1 commented Aug 19, 2026 •

edited by atlassian Bot

Loading

i-kosarev left a comment •

edited

Loading

geomin12 left a comment •

edited

Loading

geomin12 left a comment •

edited

Loading