Skip to content

Migrate MI300X runners to the Barite AMD cluster #2732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cquil11 wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Migrate MI300X runners to the Barite AMD cluster #2732

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b7594ef
feat(runners): migrate MI300X AMD runner pool
cquil11 Aug 25, 2026
ee982c8
chore: schedule MI300X migration burn-in
cquil11 Aug 25, 2026
45d3e8e
fix(amd): refresh MI300X DeepSeek V4 image
cquil11 Aug 25, 2026
c4972de
fix(runners): retry node-local image imports
cquil11 Aug 25, 2026
9baf200
fix(ci): honor burn-in duration override
cquil11 Aug 25, 2026
7fc75df
chore: remove MI300X burn-in scaffolding
cquil11 Aug 26, 2026
c7b259e
chore: keep existing MI300X images
cquil11 Aug 26, 2026
624db70
refactor(runners): simplify MI300X launcher
cquil11 Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,7 +430,7 @@ qwen3.5-fp8-mi300x-sglang-agentic-mtp:
image: lmsysorg/sglang-rocm:v0.5.16-rocm720-mi30x-20260730
model: Qwen/Qwen3.5-397B-A17B-FP8
model-prefix: qwen3.5
runner: cluster:mi300x-amds
runner: cluster:mi300x-amd
precision: fp8
framework: sglang
multinode: false
Expand Down Expand Up @@ -1516,7 +1516,7 @@ minimaxm3-fp8-mi300x-vllm-agentic-mtp:
image: vllm/vllm-openai-rocm:v0.27.1
model: MiniMaxAI/MiniMax-M3-MXFP8
model-prefix: minimaxm3
runner: cluster:mi300x-amds
runner: cluster:mi300x-amd
precision: fp8
framework: vllm
multinode: false
Expand Down
46 changes: 21 additions & 25 deletions configs/runners.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,19 +101,15 @@ labels:
- b200-nscale-slurm_7
- b200-nscale-slurm_8
mi300x:
- mi300x-amds_00
- mi300x-amds_01
- mi300x-amds_02
- mi300x-amds_03
- mi300x-amds_04
- mi300x-amds_05
- mi300x-amds_06
- mi300x-amds_07
- mi300x-amds_08
mi300x-disagg:
- mi300x-amds_06
- mi300x-amds_07
- mi300x-amds_08
- mi300x-amd_00
- mi300x-amd_01
- mi300x-amd_02
- mi300x-amd_03
- mi300x-amd_04
- mi300x-amd_05
- mi300x-amd_06
- mi300x-amd_07
- mi300x-amd_08
mi325x:
- mi325x-amds_00
- mi325x-amds_01
Expand Down Expand Up @@ -288,16 +284,16 @@ labels:
- gb300-nv_2
cluster:rtx6000pro-lat:
- rtx6000pro-lat_00
cluster:mi300x-amds:
- mi300x-amds_00
- mi300x-amds_01
- mi300x-amds_02
- mi300x-amds_03
- mi300x-amds_04
- mi300x-amds_05
- mi300x-amds_06
- mi300x-amds_07
- mi300x-amds_08
cluster:mi300x-amd:
- mi300x-amd_00
- mi300x-amd_01
- mi300x-amd_02
- mi300x-amd_03
- mi300x-amd_04
- mi300x-amd_05
- mi300x-amd_06
- mi300x-amd_07
- mi300x-amd_08
cluster:mi300x-tw:
- mi300x-tw_00
- mi300x-tw_01
Expand Down Expand Up @@ -359,8 +355,8 @@ hardware:
cluster:rtx6000pro-lat:
available-cpu-dram-mib: 1_500_000
gpus-per-node: 8
cluster:mi300x-amds:
available-cpu-dram-mib: 2_321_924
cluster:mi300x-amd:
available-cpu-dram-mib: 1_547_820
gpus-per-node: 8
cluster:mi300x-tw:
available-cpu-dram-mib: 2_322_328
Expand Down
60 changes: 60 additions & 0 deletions runners/launch_mi300x-amd.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env bash
set -euo pipefail

export HF_HUB_CACHE_MOUNT="/raid/inferencex/models/hub"
export AIPERF_MMAP_CACHE_MOUNT="/raid/inferencex/aiperf-mmap-cache"
export AIPERF_DATASET_MMAP_CACHE_DIR="/aiperf_mmap_cache"

PARTITION="compute"
SQUASH_FILE="/raid/inferencex/squash/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh"
LOCK_FILE="${SQUASH_FILE}.lock"

SPEC_SUFFIX=$([[ "${SPEC_DECODING:-}" == "mtp" ]] && printf '_mtp' || printf '')

export GPU_COUNT="${GPU_COUNT:-${TP:?TP must be set}}"

set -x

JOB_ID=$(set +o pipefail; salloc \
--partition="$PARTITION" \
--gres="gpu:$GPU_COUNT" \
--cpus-per-task=128 \
--time=180 \
--no-shell \
--job-name="$RUNNER_NAME" 2>&1 \
| tee /dev/stderr \
| grep -oP 'Granted job allocation \K[0-9]+')

if [[ -z "$JOB_ID" ]]; then
echo "ERROR: salloc failed to allocate a job" >&2
exit 1
fi

export PORT=$((40000 + (JOB_ID % 10000)))
trap 'scancel "$JOB_ID" 2>/dev/null || true' EXIT

# Use flock to serialize concurrent imports to the same node-local squash file.
srun --jobid="$JOB_ID" --job-name="$RUNNER_NAME" bash -c "
set -euo pipefail
exec 9>\"$LOCK_FILE\"
flock -w 600 9 || { echo 'Failed to acquire lock for $SQUASH_FILE' >&2; exit 1; }
if unsquashfs -l \"$SQUASH_FILE\" >/dev/null 2>&1; then
echo 'Squash file already exists and is valid, skipping import'
else
rm -f \"$SQUASH_FILE\"
enroot import -o \"$SQUASH_FILE\" docker://$IMAGE
Comment thread
cquil11 marked this conversation as resolved.
fi
"
Comment thread
cquil11 marked this conversation as resolved.

srun --jobid="$JOB_ID" \
--job-name="$RUNNER_NAME" \
--container-image="$SQUASH_FILE" \
--container-mounts="$GITHUB_WORKSPACE:/workspace/,$HF_HUB_CACHE_MOUNT:$HF_HUB_CACHE,$AIPERF_MMAP_CACHE_MOUNT:/aiperf_mmap_cache,/dev/kfd:/dev/kfd,/dev/dri:/dev/dri" \
--container-writable \
--container-remap-root \
--container-workdir=/workspace/ \
--no-container-entrypoint \
--export=ALL \
bash "benchmarks/single_node/${SCENARIO_SUBDIR}${EXP_NAME%%_*}_${PRECISION}_mi300x${SPEC_SUFFIX}.sh"

scancel "$JOB_ID"
112 changes: 0 additions & 112 deletions runners/launch_mi300x-amds.sh
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion utils/matrix_logic/test_generate_sweep_configs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ def sample_runner_config():
"cluster:h200-dgxc": {"available-cpu-dram-mib": 1471356, "gpus-per-node": 8},
"cluster:b200-dgxc": {"available-cpu-dram-mib": 3774874, "gpus-per-node": 8},
"cluster:b300-nv": {"available-cpu-dram-mib": 2964436, "gpus-per-node": 8},
"cluster:mi300x-amds": {"available-cpu-dram-mib": 2321924, "gpus-per-node": 8},
"cluster:mi300x-amd": {"available-cpu-dram-mib": 1547820, "gpus-per-node": 8},
"cluster:mi355x-amds": {"available-cpu-dram-mib": 3095781, "gpus-per-node": 8},
"cluster:gb200-nv": {"available-cpu-dram-mib": 860160, "gpus-per-node": 4},
},
Expand Down
2 changes: 1 addition & 1 deletion utils/matrix_logic/test_validation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ def valid_runner_config():
"cluster:h100-dgxc": {"available-cpu-dram-mib": 2063837, "gpus-per-node": 8},
"cluster:h200-dgxc": {"available-cpu-dram-mib": 1471356, "gpus-per-node": 8},
"cluster:b200-dgxc": {"available-cpu-dram-mib": 3774874, "gpus-per-node": 8},
"cluster:mi300x-amds": {"available-cpu-dram-mib": 2321924, "gpus-per-node": 8},
"cluster:mi300x-amd": {"available-cpu-dram-mib": 1547820, "gpus-per-node": 8},
"cluster:gb200-nv": {"available-cpu-dram-mib": 860160, "gpus-per-node": 4},
},
}
Expand Down
2 changes: 1 addition & 1 deletion utils/test_validate_reusable_sweep_artifacts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -402,7 +402,7 @@ def test_eval_validation_uses_logical_runner_from_metadata(
tmp_path,
64,
logical_runner="mi300x",
physical_runner="mi300x-amds_04",
physical_runner="mi300x-amd_04",
)

assert validate_eval_artifacts(tmp_path) == []
Expand Down
Loading