[DO NOT MERGE] [ROCm] device sync on pytorch module exit#3369
Draft
dnikolaev-amd wants to merge 1 commit into
Draft
[DO NOT MERGE] [ROCm] device sync on pytorch module exit#3369dnikolaev-amd wants to merge 1 commit into
dnikolaev-amd wants to merge 1 commit into
ROCm Repo Management API / Jenkins
failed
Jun 25, 2026 in 12h 28m 34s
Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
Build log truncated.
[2026-06-24T21:24:20.234Z] Allocatable: FALSE
[2026-06-24T21:24:20.234Z] Alloc Granule: 0KB
[2026-06-24T21:24:20.234Z] Alloc Recommended Granule:0KB
[2026-06-24T21:24:20.234Z] Alloc Alignment: 0KB
[2026-06-24T21:24:20.234Z] Accessible by all: FALSE
[2026-06-24T21:24:20.234Z] ISA Info:
[2026-06-24T21:24:20.234Z] ISA 1
[2026-06-24T21:24:20.234Z] Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
[2026-06-24T21:24:20.234Z] Machine Models: HSA_MACHINE_MODEL_LARGE
[2026-06-24T21:24:20.234Z] Profiles: HSA_PROFILE_BASE
[2026-06-24T21:24:20.234Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.234Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.234Z] Fast f16: TRUE
[2026-06-24T21:24:20.234Z] Workgroup Max Size: 1024(0x400)
[2026-06-24T21:24:20.234Z] Workgroup Max Size per Dimension:
[2026-06-24T21:24:20.234Z] x 1024(0x400)
[2026-06-24T21:24:20.234Z] y 1024(0x400)
[2026-06-24T21:24:20.234Z] z 1024(0x400)
[2026-06-24T21:24:20.234Z] Grid Max Size: 4294967295(0xffffffff)
[2026-06-24T21:24:20.234Z] Grid Max Size per Dimension:
[2026-06-24T21:24:20.234Z] x 2147483647(0x7fffffff)
[2026-06-24T21:24:20.234Z] y 65535(0xffff)
[2026-06-24T21:24:20.234Z] z 65535(0xffff)
[2026-06-24T21:24:20.234Z] FBarrier Max Size: 32
[2026-06-24T21:24:20.234Z] *******
[2026-06-24T21:24:20.234Z] Agent 7
[2026-06-24T21:24:20.234Z] *******
[2026-06-24T21:24:20.234Z] Name: gfx90a
[2026-06-24T21:24:20.234Z] Uuid: GPU-baaba295e71e5f1d
[2026-06-24T21:24:20.234Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:20.234Z] Vendor Name: AMD
[2026-06-24T21:24:20.234Z] Feature: KERNEL_DISPATCH
[2026-06-24T21:24:20.234Z] Profile: BASE_PROFILE
[2026-06-24T21:24:20.234Z] Float Round Mode: NEAR
[2026-06-24T21:24:20.234Z] Max Queue Number: 128(0x80)
[2026-06-24T21:24:20.234Z] Queue Min Size: 64(0x40)
[2026-06-24T21:24:20.234Z] Queue Max Size: 131072(0x20000)
[2026-06-24T21:24:20.234Z] Queue Type: MULTI
[2026-06-24T21:24:20.234Z] Node: 6
[2026-06-24T21:24:20.234Z] Device Type: GPU
[2026-06-24T21:24:20.234Z] Cache Info:
[2026-06-24T21:24:20.234Z] L1: 16(0x10) KB
[2026-06-24T21:24:20.234Z] L2: 8192(0x2000) KB
[2026-06-24T21:24:20.234Z] Chip ID: 29708(0x740c)
[2026-06-24T21:24:20.234Z] ASIC Revision: 1(0x1)
[2026-06-24T21:24:20.234Z] Cacheline Size: 128(0x80)
[2026-06-24T21:24:20.234Z] Max Clock Freq. (MHz): 1700
[2026-06-24T21:24:20.234Z] BDFID: 44544
[2026-06-24T21:24:20.234Z] Internal Node ID: 6
[2026-06-24T21:24:20.234Z] Compute Unit: 104
[2026-06-24T21:24:20.234Z] SIMDs per CU: 4
[2026-06-24T21:24:20.234Z] Shader Engines: 8
[2026-06-24T21:24:20.234Z] Shader Arrs. per Eng.: 1
[2026-06-24T21:24:20.234Z] WatchPts on Addr. Ranges:4
[2026-06-24T21:24:20.234Z] Coherent Host Access: FALSE
[2026-06-24T21:24:20.234Z] Memory Properties:
[2026-06-24T21:24:20.234Z] Features: KERNEL_DISPATCH
[2026-06-24T21:24:20.234Z] Fast F16 Operation: TRUE
[2026-06-24T21:24:20.234Z] Wavefront Size: 64(0x40)
[2026-06-24T21:24:20.234Z] Workgroup Max Size: 1024(0x400)
[2026-06-24T21:24:20.234Z] Workgroup Max Size per Dimension:
[2026-06-24T21:24:20.234Z] x 1024(0x400)
[2026-06-24T21:24:20.234Z] y 1024(0x400)
[2026-06-24T21:24:20.234Z] z 1024(0x400)
[2026-06-24T21:24:20.234Z] Max Waves Per CU: 32(0x20)
[2026-06-24T21:24:20.234Z] Max Work-item Per CU: 2048(0x800)
[2026-06-24T21:24:20.234Z] Grid Max Size: 4294967295(0xffffffff)
[2026-06-24T21:24:20.234Z] Grid Max Size per Dimension:
[2026-06-24T21:24:20.234Z] x 2147483647(0x7fffffff)
[2026-06-24T21:24:20.234Z] y 65535(0xffff)
[2026-06-24T21:24:20.234Z] z 65535(0xffff)
[2026-06-24T21:24:20.234Z] Max fbarriers/Workgrp: 32
[2026-06-24T21:24:20.234Z] Packet Processor uCode:: 100
[2026-06-24T21:24:20.234Z] SDMA engine uCode:: 9
[2026-06-24T21:24:20.234Z] IOMMU Support:: None
[2026-06-24T21:24:20.234Z] Pool Info:
[2026-06-24T21:24:20.234Z] Pool 1
[2026-06-24T21:24:20.234Z] Segment: GLOBAL; FLAGS: COARSE GRAINED
[2026-06-24T21:24:20.234Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.234Z] Allocatable: TRUE
[2026-06-24T21:24:20.234Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.234Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.234Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.234Z] Accessible by all: FALSE
[2026-06-24T21:24:20.234Z] Pool 2
[2026-06-24T21:24:20.234Z] Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
[2026-06-24T21:24:20.234Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.234Z] Allocatable: TRUE
[2026-06-24T21:24:20.234Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.234Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.234Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.234Z] Accessible by all: FALSE
[2026-06-24T21:24:20.234Z] Pool 3
[2026-06-24T21:24:20.234Z] Segment: GLOBAL; FLAGS: FINE GRAINED
[2026-06-24T21:24:20.234Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.234Z] Allocatable: TRUE
[2026-06-24T21:24:20.234Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.234Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.234Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.234Z] Accessible by all: FALSE
[2026-06-24T21:24:20.234Z] Pool 4
[2026-06-24T21:24:20.234Z] Segment: GROUP
[2026-06-24T21:24:20.234Z] Size: 64(0x40) KB
[2026-06-24T21:24:20.234Z] Allocatable: FALSE
[2026-06-24T21:24:20.235Z] Alloc Granule: 0KB
[2026-06-24T21:24:20.235Z] Alloc Recommended Granule:0KB
[2026-06-24T21:24:20.235Z] Alloc Alignment: 0KB
[2026-06-24T21:24:20.235Z] Accessible by all: FALSE
[2026-06-24T21:24:20.235Z] ISA Info:
[2026-06-24T21:24:20.235Z] ISA 1
[2026-06-24T21:24:20.235Z] Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
[2026-06-24T21:24:20.235Z] Machine Models: HSA_MACHINE_MODEL_LARGE
[2026-06-24T21:24:20.235Z] Profiles: HSA_PROFILE_BASE
[2026-06-24T21:24:20.235Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.235Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.235Z] Fast f16: TRUE
[2026-06-24T21:24:20.235Z] Workgroup Max Size: 1024(0x400)
[2026-06-24T21:24:20.235Z] Workgroup Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 1024(0x400)
[2026-06-24T21:24:20.235Z] y 1024(0x400)
[2026-06-24T21:24:20.235Z] z 1024(0x400)
[2026-06-24T21:24:20.235Z] Grid Max Size: 4294967295(0xffffffff)
[2026-06-24T21:24:20.235Z] Grid Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 2147483647(0x7fffffff)
[2026-06-24T21:24:20.235Z] y 65535(0xffff)
[2026-06-24T21:24:20.235Z] z 65535(0xffff)
[2026-06-24T21:24:20.235Z] FBarrier Max Size: 32
[2026-06-24T21:24:20.235Z] *******
[2026-06-24T21:24:20.235Z] Agent 8
[2026-06-24T21:24:20.235Z] *******
[2026-06-24T21:24:20.235Z] Name: gfx90a
[2026-06-24T21:24:20.235Z] Uuid: GPU-bde9f33df372dcef
[2026-06-24T21:24:20.235Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:20.235Z] Vendor Name: AMD
[2026-06-24T21:24:20.235Z] Feature: KERNEL_DISPATCH
[2026-06-24T21:24:20.235Z] Profile: BASE_PROFILE
[2026-06-24T21:24:20.235Z] Float Round Mode: NEAR
[2026-06-24T21:24:20.235Z] Max Queue Number: 128(0x80)
[2026-06-24T21:24:20.235Z] Queue Min Size: 64(0x40)
[2026-06-24T21:24:20.235Z] Queue Max Size: 131072(0x20000)
[2026-06-24T21:24:20.235Z] Queue Type: MULTI
[2026-06-24T21:24:20.235Z] Node: 7
[2026-06-24T21:24:20.235Z] Device Type: GPU
[2026-06-24T21:24:20.235Z] Cache Info:
[2026-06-24T21:24:20.235Z] L1: 16(0x10) KB
[2026-06-24T21:24:20.235Z] L2: 8192(0x2000) KB
[2026-06-24T21:24:20.235Z] Chip ID: 29708(0x740c)
[2026-06-24T21:24:20.235Z] ASIC Revision: 1(0x1)
[2026-06-24T21:24:20.235Z] Cacheline Size: 128(0x80)
[2026-06-24T21:24:20.235Z] Max Clock Freq. (MHz): 1700
[2026-06-24T21:24:20.235Z] BDFID: 45824
[2026-06-24T21:24:20.235Z] Internal Node ID: 7
[2026-06-24T21:24:20.235Z] Compute Unit: 104
[2026-06-24T21:24:20.235Z] SIMDs per CU: 4
[2026-06-24T21:24:20.235Z] Shader Engines: 8
[2026-06-24T21:24:20.235Z] Shader Arrs. per Eng.: 1
[2026-06-24T21:24:20.235Z] WatchPts on Addr. Ranges:4
[2026-06-24T21:24:20.235Z] Coherent Host Access: FALSE
[2026-06-24T21:24:20.235Z] Memory Properties:
[2026-06-24T21:24:20.235Z] Features: KERNEL_DISPATCH
[2026-06-24T21:24:20.235Z] Fast F16 Operation: TRUE
[2026-06-24T21:24:20.235Z] Wavefront Size: 64(0x40)
[2026-06-24T21:24:20.235Z] Workgroup Max Size: 1024(0x400)
[2026-06-24T21:24:20.235Z] Workgroup Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 1024(0x400)
[2026-06-24T21:24:20.235Z] y 1024(0x400)
[2026-06-24T21:24:20.235Z] z 1024(0x400)
[2026-06-24T21:24:20.235Z] Max Waves Per CU: 32(0x20)
[2026-06-24T21:24:20.235Z] Max Work-item Per CU: 2048(0x800)
[2026-06-24T21:24:20.235Z] Grid Max Size: 4294967295(0xffffffff)
[2026-06-24T21:24:20.235Z] Grid Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 2147483647(0x7fffffff)
[2026-06-24T21:24:20.235Z] y 65535(0xffff)
[2026-06-24T21:24:20.235Z] z 65535(0xffff)
[2026-06-24T21:24:20.235Z] Max fbarriers/Workgrp: 32
[2026-06-24T21:24:20.235Z] Packet Processor uCode:: 100
[2026-06-24T21:24:20.235Z] SDMA engine uCode:: 9
[2026-06-24T21:24:20.235Z] IOMMU Support:: None
[2026-06-24T21:24:20.235Z] Pool Info:
[2026-06-24T21:24:20.235Z] Pool 1
[2026-06-24T21:24:20.235Z] Segment: GLOBAL; FLAGS: COARSE GRAINED
[2026-06-24T21:24:20.235Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.235Z] Allocatable: TRUE
[2026-06-24T21:24:20.235Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.235Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.235Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.235Z] Accessible by all: FALSE
[2026-06-24T21:24:20.235Z] Pool 2
[2026-06-24T21:24:20.235Z] Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
[2026-06-24T21:24:20.235Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.235Z] Allocatable: TRUE
[2026-06-24T21:24:20.235Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.235Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.235Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.235Z] Accessible by all: FALSE
[2026-06-24T21:24:20.235Z] Pool 3
[2026-06-24T21:24:20.235Z] Segment: GLOBAL; FLAGS: FINE GRAINED
[2026-06-24T21:24:20.235Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.235Z] Allocatable: TRUE
[2026-06-24T21:24:20.235Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.235Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.235Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.235Z] Accessible by all: FALSE
[2026-06-24T21:24:20.235Z] Pool 4
[2026-06-24T21:24:20.235Z] Segment: GROUP
[2026-06-24T21:24:20.235Z] Size: 64(0x40) KB
[2026-06-24T21:24:20.235Z] Allocatable: FALSE
[2026-06-24T21:24:20.235Z] Alloc Granule: 0KB
[2026-06-24T21:24:20.235Z] Alloc Recommended Granule:0KB
[2026-06-24T21:24:20.235Z] Alloc Alignment: 0KB
[2026-06-24T21:24:20.235Z] Accessible by all: FALSE
[2026-06-24T21:24:20.235Z] ISA Info:
[2026-06-24T21:24:20.235Z] ISA 1
[2026-06-24T21:24:20.235Z] Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
[2026-06-24T21:24:20.235Z] Machine Models: HSA_MACHINE_MODEL_LARGE
[2026-06-24T21:24:20.235Z] Profiles: HSA_PROFILE_BASE
[2026-06-24T21:24:20.235Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.235Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.235Z] Fast f16: TRUE
[2026-06-24T21:24:20.235Z] Workgroup Max Size: 1024(0x400)
[2026-06-24T21:24:20.235Z] Workgroup Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 1024(0x400)
[2026-06-24T21:24:20.235Z] y 1024(0x400)
[2026-06-24T21:24:20.235Z] z 1024(0x400)
[2026-06-24T21:24:20.235Z] Grid Max Size: 4294967295(0xffffffff)
[2026-06-24T21:24:20.235Z] Grid Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 2147483647(0x7fffffff)
[2026-06-24T21:24:20.235Z] y 65535(0xffff)
[2026-06-24T21:24:20.235Z] z 65535(0xffff)
[2026-06-24T21:24:20.235Z] FBarrier Max Size: 32
[2026-06-24T21:24:20.235Z] *******
[2026-06-24T21:24:20.235Z] Agent 9
[2026-06-24T21:24:20.235Z] *******
[2026-06-24T21:24:20.235Z] Name: gfx90a
[2026-06-24T21:24:20.235Z] Uuid: GPU-61081c54efa4d731
[2026-06-24T21:24:20.235Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:20.235Z] Vendor Name: AMD
[2026-06-24T21:24:20.235Z] Feature: KERNEL_DISPATCH
[2026-06-24T21:24:20.235Z] Profile: BASE_PROFILE
[2026-06-24T21:24:20.235Z] Float Round Mode: NEAR
[2026-06-24T21:24:20.235Z] Max Queue Number: 128(0x80)
[2026-06-24T21:24:20.235Z] Queue Min Size: 64(0x40)
[2026-06-24T21:24:20.235Z] Queue Max Size: 131072(0x20000)
[2026-06-24T21:24:20.235Z] Queue Type: MULTI
[2026-06-24T21:24:20.235Z] Node: 8
[2026-06-24T21:24:20.235Z] Device Type: GPU
[2026-06-24T21:24:20.235Z] Cache Info:
[2026-06-24T21:24:20.235Z] L1: 16(0x10) KB
[2026-06-24T21:24:20.235Z] L2: 8192(0x2000) KB
[2026-06-24T21:24:20.235Z] Chip ID: 29708(0x740c)
[2026-06-24T21:24:20.235Z] ASIC Revision: 1(0x1)
[2026-06-24T21:24:20.235Z] Cacheline Size: 128(0x80)
[2026-06-24T21:24:20.235Z] Max Clock Freq. (MHz): 1700
[2026-06-24T21:24:20.235Z] BDFID: 36352
[2026-06-24T21:24:20.235Z] Internal Node ID: 8
[2026-06-24T21:24:20.235Z] Compute Unit: 104
[2026-06-24T21:24:20.235Z] SIMDs per CU: 4
[2026-06-24T21:24:20.235Z] Shader Engines: 8
[2026-06-24T21:24:20.235Z] Shader Arrs. per Eng.: 1
[2026-06-24T21:24:20.235Z] WatchPts on Addr. Ranges:4
[2026-06-24T21:24:20.235Z] Coherent Host Access: FALSE
[2026-06-24T21:24:20.235Z] Memory Properties:
[2026-06-24T21:24:20.235Z] Features: KERNEL_DISPATCH
[2026-06-24T21:24:20.235Z] Fast F16 Operation: TRUE
[2026-06-24T21:24:20.235Z] Wavefront Size: 64(0x40)
[2026-06-24T21:24:20.235Z] Workgroup Max Size: 1024(0x400)
[2026-06-24T21:24:20.235Z] Workgroup Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 1024(0x400)
[2026-06-24T21:24:20.235Z] y 1024(0x400)
[2026-06-24T21:24:20.235Z] z 1024(0x400)
[2026-06-24T21:24:20.235Z] Max Waves Per CU: 32(0x20)
[2026-06-24T21:24:20.235Z] Max Work-item Per CU: 2048(0x800)
[2026-06-24T21:24:20.235Z] Grid Max Size: 4294967295(0xffffffff)
[2026-06-24T21:24:20.235Z] Grid Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 2147483647(0x7fffffff)
[2026-06-24T21:24:20.235Z] y 65535(0xffff)
[2026-06-24T21:24:20.235Z] z 65535(0xffff)
[2026-06-24T21:24:20.235Z] Max fbarriers/Workgrp: 32
[2026-06-24T21:24:20.235Z] Packet Processor uCode:: 100
[2026-06-24T21:24:20.235Z] SDMA engine uCode:: 9
[2026-06-24T21:24:20.235Z] IOMMU Support:: None
[2026-06-24T21:24:20.235Z] Pool Info:
[2026-06-24T21:24:20.235Z] Pool 1
[2026-06-24T21:24:20.235Z] Segment: GLOBAL; FLAGS: COARSE GRAINED
[2026-06-24T21:24:20.235Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.235Z] Allocatable: TRUE
[2026-06-24T21:24:20.235Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.235Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.235Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.235Z] Accessible by all: FALSE
[2026-06-24T21:24:20.235Z] Pool 2
[2026-06-24T21:24:20.235Z] Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
[2026-06-24T21:24:20.235Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.235Z] Allocatable: TRUE
[2026-06-24T21:24:20.235Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.235Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.235Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.235Z] Accessible by all: FALSE
[2026-06-24T21:24:20.235Z] Pool 3
[2026-06-24T21:24:20.235Z] Segment: GLOBAL; FLAGS: FINE GRAINED
[2026-06-24T21:24:20.235Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.235Z] Allocatable: TRUE
[2026-06-24T21:24:20.235Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.235Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.235Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.235Z] Accessible by all: FALSE
[2026-06-24T21:24:20.235Z] Pool 4
[2026-06-24T21:24:20.235Z] Segment: GROUP
[2026-06-24T21:24:20.235Z] Size: 64(0x40) KB
[2026-06-24T21:24:20.235Z] Allocatable: FALSE
[2026-06-24T21:24:20.235Z] Alloc Granule: 0KB
[2026-06-24T21:24:20.235Z] Alloc Recommended Granule:0KB
[2026-06-24T21:24:20.235Z] Alloc Alignment: 0KB
[2026-06-24T21:24:20.235Z] Accessible by all: FALSE
[2026-06-24T21:24:20.235Z] ISA Info:
[2026-06-24T21:24:20.235Z] ISA 1
[2026-06-24T21:24:20.235Z] Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
[2026-06-24T21:24:20.235Z] Machine Models: HSA_MACHINE_MODEL_LARGE
[2026-06-24T21:24:20.235Z] Profiles: HSA_PROFILE_BASE
[2026-06-24T21:24:20.235Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.235Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.235Z] Fast f16: TRUE
[2026-06-24T21:24:20.235Z] Workgroup Max Size: 1024(0x400)
[2026-06-24T21:24:20.235Z] Workgroup Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 1024(0x400)
[2026-06-24T21:24:20.235Z] y 1024(0x400)
[2026-06-24T21:24:20.235Z] z 1024(0x400)
[2026-06-24T21:24:20.235Z] Grid Max Size: 4294967295(0xffffffff)
[2026-06-24T21:24:20.235Z] Grid Max Size per Dimension:
[2026-06-24T21:24:20.235Z] x 2147483647(0x7fffffff)
[2026-06-24T21:24:20.235Z] y 65535(0xffff)
[2026-06-24T21:24:20.235Z] z 65535(0xffff)
[2026-06-24T21:24:20.235Z] FBarrier Max Size: 32
[2026-06-24T21:24:20.235Z] *******
[2026-06-24T21:24:20.235Z] Agent 10
[2026-06-24T21:24:20.235Z] *******
[2026-06-24T21:24:20.235Z] Name: gfx90a
[2026-06-24T21:24:20.235Z] Uuid: GPU-fc77cd7dd4e7131c
[2026-06-24T21:24:20.235Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:20.235Z] Vendor Name: AMD
[2026-06-24T21:24:20.235Z] Feature: KERNEL_DISPATCH
[2026-06-24T21:24:20.235Z] Profile: BASE_PROFILE
[2026-06-24T21:24:20.235Z] Float Round Mode: NEAR
[2026-06-24T21:24:20.235Z] Max Queue Number: 128(0x80)
[2026-06-24T21:24:20.235Z] Queue Min Size: 64(0x40)
[2026-06-24T21:24:20.235Z] Queue Max Size: 131072(0x20000)
[2026-06-24T21:24:20.235Z] Queue Type: MULTI
[2026-06-24T21:24:20.235Z] Node: 9
[2026-06-24T21:24:20.235Z] Device Type: GPU
[2026-06-24T21:24:20.235Z] Cache Info:
[2026-06-24T21:24:20.235Z] L1: 16(0x10) KB
[2026-06-24T21:24:20.235Z] L2: 8192(0x2000) KB
[2026-06-24T21:24:20.235Z] Chip ID: 29708(0x740c)
[2026-06-24T21:24:20.235Z] ASIC Revision: 1(0x1)
[2026-06-24T21:24:20.276Z] Cacheline Size: 128(0x80)
[2026-06-24T21:24:20.276Z] Max Clock Freq. (MHz): 1700
[2026-06-24T21:24:20.276Z] BDFID: 37632
[2026-06-24T21:24:20.276Z] Internal Node ID: 9
[2026-06-24T21:24:20.276Z] Compute Unit: 104
[2026-06-24T21:24:20.276Z] SIMDs per CU: 4
[2026-06-24T21:24:20.276Z] Shader Engines: 8
[2026-06-24T21:24:20.276Z] Shader Arrs. per Eng.: 1
[2026-06-24T21:24:20.276Z] WatchPts on Addr. Ranges:4
[2026-06-24T21:24:20.276Z] Coherent Host Access: FALSE
[2026-06-24T21:24:20.276Z] Memory Properties:
[2026-06-24T21:24:20.276Z] Features: KERNEL_DISPATCH
[2026-06-24T21:24:20.276Z] Fast F16 Operation: TRUE
[2026-06-24T21:24:20.276Z] Wavefront Size: 64(0x40)
[2026-06-24T21:24:20.276Z] Workgroup Max Size: 1024(0x400)
[2026-06-24T21:24:20.276Z] Workgroup Max Size per Dimension:
[2026-06-24T21:24:20.276Z] x 1024(0x400)
[2026-06-24T21:24:20.276Z] y 1024(0x400)
[2026-06-24T21:24:20.276Z] z 1024(0x400)
[2026-06-24T21:24:20.276Z] Max Waves Per CU: 32(0x20)
[2026-06-24T21:24:20.276Z] Max Work-item Per CU: 2048(0x800)
[2026-06-24T21:24:20.276Z] Grid Max Size: 4294967295(0xffffffff)
[2026-06-24T21:24:20.276Z] Grid Max Size per Dimension:
[2026-06-24T21:24:20.276Z] x 2147483647(0x7fffffff)
[2026-06-24T21:24:20.276Z] y 65535(0xffff)
[2026-06-24T21:24:20.276Z] z 65535(0xffff)
[2026-06-24T21:24:20.276Z] Max fbarriers/Workgrp: 32
[2026-06-24T21:24:20.276Z] Packet Processor uCode:: 100
[2026-06-24T21:24:20.276Z] SDMA engine uCode:: 9
[2026-06-24T21:24:20.276Z] IOMMU Support:: None
[2026-06-24T21:24:20.276Z] Pool Info:
[2026-06-24T21:24:20.276Z] Pool 1
[2026-06-24T21:24:20.276Z] Segment: GLOBAL; FLAGS: COARSE GRAINED
[2026-06-24T21:24:20.276Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.276Z] Allocatable: TRUE
[2026-06-24T21:24:20.276Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.276Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.276Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.276Z] Accessible by all: FALSE
[2026-06-24T21:24:20.276Z] Pool 2
[2026-06-24T21:24:20.276Z] Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
[2026-06-24T21:24:20.276Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.276Z] Allocatable: TRUE
[2026-06-24T21:24:20.276Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.276Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.276Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.276Z] Accessible by all: FALSE
[2026-06-24T21:24:20.276Z] Pool 3
[2026-06-24T21:24:20.276Z] Segment: GLOBAL; FLAGS: FINE GRAINED
[2026-06-24T21:24:20.276Z] Size: 67092480(0x3ffc000) KB
[2026-06-24T21:24:20.276Z] Allocatable: TRUE
[2026-06-24T21:24:20.276Z] Alloc Granule: 4KB
[2026-06-24T21:24:20.276Z] Alloc Recommended Granule:2048KB
[2026-06-24T21:24:20.276Z] Alloc Alignment: 4KB
[2026-06-24T21:24:20.276Z] Accessible by all: FALSE
[2026-06-24T21:24:20.276Z] Pool 4
[2026-06-24T21:24:20.276Z] Segment: GROUP
[2026-06-24T21:24:20.276Z] Size: 64(0x40) KB
[2026-06-24T21:24:20.276Z] Allocatable: FALSE
[2026-06-24T21:24:20.276Z] Alloc Granule: 0KB
[2026-06-24T21:24:20.276Z] Alloc Recommended Granule:0KB
[2026-06-24T21:24:20.276Z] Alloc Alignment: 0KB
[2026-06-24T21:24:20.276Z] Accessible by all: FALSE
[2026-06-24T21:24:20.276Z] ISA Info:
[2026-06-24T21:24:20.276Z] ISA 1
[2026-06-24T21:24:20.276Z] Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
[2026-06-24T21:24:20.276Z] Machine Models: HSA_MACHINE_MODEL_LARGE
[2026-06-24T21:24:20.276Z] Profiles: HSA_PROFILE_BASE
[2026-06-24T21:24:20.276Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.276Z] Default Rounding Mode: NEAR
[2026-06-24T21:24:20.276Z] Fast f16: TRUE
[2026-06-24T21:24:20.276Z] Workgroup Max Size: 1024(0x400)
[2026-06-24T21:24:20.276Z] Workgroup Max Size per Dimension:
[2026-06-24T21:24:20.276Z] x 1024(0x400)
[2026-06-24T21:24:20.276Z] y 1024(0x400)
[2026-06-24T21:24:20.276Z] z 1024(0x400)
[2026-06-24T21:24:20.276Z] Grid Max Size: 4294967295(0xffffffff)
[2026-06-24T21:24:20.276Z] Grid Max Size per Dimension:
[2026-06-24T21:24:20.276Z] x 2147483647(0x7fffffff)
[2026-06-24T21:24:20.276Z] y 65535(0xffff)
[2026-06-24T21:24:20.276Z] z 65535(0xffff)
[2026-06-24T21:24:20.276Z] FBarrier Max Size: 32
[2026-06-24T21:24:20.276Z] *** Done ***
[2026-06-24T21:24:20.276Z] + rocminfo
[2026-06-24T21:24:20.276Z] + grep -E 'Name:.*\sgfx|Marketing'
[2026-06-24T21:24:22.513Z] Marketing Name: AMD EPYC 7573X 32-Core Processor
[2026-06-24T21:24:22.513Z] Marketing Name: AMD EPYC 7573X 32-Core Processor
[2026-06-24T21:24:22.513Z] Name: gfx90a
[2026-06-24T21:24:22.513Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:22.513Z] Name: gfx90a
[2026-06-24T21:24:22.513Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:22.513Z] Name: gfx90a
[2026-06-24T21:24:22.513Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:22.513Z] Name: gfx90a
[2026-06-24T21:24:22.513Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:22.513Z] Name: gfx90a
[2026-06-24T21:24:22.513Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:22.513Z] Name: gfx90a
[2026-06-24T21:24:22.513Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:22.513Z] Name: gfx90a
[2026-06-24T21:24:22.513Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:22.513Z] Name: gfx90a
[2026-06-24T21:24:22.513Z] Marketing Name: AMD Instinct MI250X / MI250
[2026-06-24T21:24:22.513Z] + MAYBE_ROCM=rocm/
[2026-06-24T21:24:22.513Z] + [[ pytorch-linux-noble-rocm7.2.4-py3.12-test2 == *xpu* ]]
[2026-06-24T21:24:22.513Z] + export PATH=/root/.local/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/opt/cache/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[2026-06-24T21:24:22.513Z] + PATH=/root/.local/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/opt/cache/bin:/opt/rocm/bin:/opt/rocm/llvm/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[2026-06-24T21:24:22.513Z] + [[ pytorch-linux-noble-rocm7.2.4-py3.12-test2 == *aarch64* ]]
[2026-06-24T21:24:22.513Z] + [[ pytorch-linux-noble-rocm7.2.4-py3.12-test2 == *asan* ]]
[2026-06-24T21:24:22.513Z] + [[ pytorch-linux-noble-rocm7.2.4-py3.12-test2 == *-debug* ]]
[2026-06-24T21:24:22.513Z] + echo 'We are not in debug mode: pytorch-linux-noble-rocm7.2.4-py3.12-test2. Expect the assertion to pass'
[2026-06-24T21:24:22.513Z] We are not in debug mode: pytorch-linux-noble-rocm7.2.4-py3.12-test2. Expect the assertion to pass
[2026-06-24T21:24:22.513Z] + cd test
[2026-06-24T21:24:22.513Z] + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
[2026-06-24T21:24:22.513Z] + [[ default == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
[2026-06-24T21:24:22.513Z] + [[ default == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
[2026-06-24T21:24:22.513Z] + DYNAMO_BENCHMARK_FLAGS=()
[2026-06-24T21:24:22.513Z] + [[ default == *pr_time_benchmarks* ]]
[2026-06-24T21:24:22.513Z] + [[ default == *dynamo_eager* ]]
[2026-06-24T21:24:22.513Z] + [[ default == *aot_eager* ]]
[2026-06-24T21:24:22.513Z] + [[ default == *aot_inductor* ]]
[2026-06-24T21:24:22.513Z] + [[ default == *max_autotune_inductor* ]]
[2026-06-24T21:24:22.513Z] + [[ default == *inductor* ]]
[2026-06-24T21:24:22.513Z] + [[ default == *dynamic* ]]
[2026-06-24T21:24:22.513Z] + [[ default == *cpu* ]]
[2026-06-24T21:24:22.513Z] + [[ default == *xpu* ]]
[2026-06-24T21:24:22.513Z] + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
[2026-06-24T21:24:22.513Z] + [[ pytorch-linux-noble-rocm7.2.4-py3.12-test2 == *libtorch* ]]
[2026-06-24T21:24:22.513Z] + cd test
[2026-06-24T21:24:22.513Z] + python -c 'import torch; print(torch.__config__.show())'
[2026-06-24T21:24:24.000Z] PyTorch built with:
[2026-06-24T21:24:24.000Z] - GCC 13.3
[2026-06-24T21:24:24.000Z] - C++ Version: 202002
[2026-06-24T21:24:24.000Z] - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
[2026-06-24T21:24:24.000Z] - Intel(R) MKL-DNN v3.11.2 (Git Hash 03c022d3ffdcee958cfacbe720048e725fdf644c)
[2026-06-24T21:24:24.000Z] - OpenMP 201511 (a.k.a. OpenMP 4.5)
[2026-06-24T21:24:24.000Z] - LAPACK is enabled (usually provided by MKL)
[2026-06-24T21:24:24.000Z] - NNPACK is enabled
[2026-06-24T21:24:24.000Z] - CPU capability usage: AVX2
[2026-06-24T21:24:24.000Z] - HIP Runtime 7.2.53211
[2026-06-24T21:24:24.000Z] - MIOpen 3.5.1
[2026-06-24T21:24:24.000Z] - Magma 2.9.0
[2026-06-24T21:24:24.001Z] - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=017382841d6df6a8704bce2240fe25f193dee5e8, CUDA_FLAGS= -DLIBCUDACXX_ENABLE_SIMPLIFIED_COMPLEX_OPERATIONS -Xfatbin -compress-all -Wno-deprecated-gpu-targets --expt-extended-lambda -DCUB_WRAPPED_NAMESPACE=at_cuda_detail -DDISABLE_CUSPARSE_DEPRECATED -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -DC10_NODEPRECATED, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_MSLK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.12.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF,
[2026-06-24T21:24:24.001Z]
[2026-06-24T21:24:24.382Z] + cd test
[2026-06-24T21:24:24.382Z] + python -c 'import torch; print(torch.__config__.parallel_info())'
[2026-06-24T21:24:26.314Z] ATen/Parallel:
[2026-06-24T21:24:26.314Z] at::get_num_threads() : 64
[2026-06-24T21:24:26.314Z] at::get_num_interop_threads() : 64
[2026-06-24T21:24:26.314Z] OpenMP 201511 (a.k.a. OpenMP 4.5)
[2026-06-24T21:24:26.314Z] omp_get_max_threads() : 64
[2026-06-24T21:24:26.314Z] Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
[2026-06-24T21:24:26.314Z] mkl_get_max_threads() : 64
[2026-06-24T21:24:26.314Z] Intel(R) MKL-DNN v3.11.2 (Git Hash 03c022d3ffdcee958cfacbe720048e725fdf644c)
[2026-06-24T21:24:26.314Z] std::thread::hardware_concurrency() : 64
[2026-06-24T21:24:26.314Z] Environment variables:
[2026-06-24T21:24:26.314Z] OMP_NUM_THREADS : [not set]
[2026-06-24T21:24:26.314Z] MKL_NUM_THREADS : [not set]
[2026-06-24T21:24:26.314Z] ATen parallel backend: OpenMP
[2026-06-24T21:24:26.314Z]
[2026-06-24T21:24:26.314Z] + [[ default == \o\n\n\x ]]
[2026-06-24T21:24:26.314Z] + [[ default == *numpy_2* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *backward* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *libtorch_agnostic_targetting* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *xla* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *vllm* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *torchtitan* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *executorch* ]]
[2026-06-24T21:24:26.314Z] + [[ default == \j\i\t\_\l\e\g\a\c\y ]]
[2026-06-24T21:24:26.314Z] + [[ default == \q\u\a\n\t\i\z\a\t\i\o\n ]]
[2026-06-24T21:24:26.314Z] + [[ pytorch-linux-noble-rocm7.2.4-py3.12-test2 == *libtorch* ]]
[2026-06-24T21:24:26.314Z] + [[ default == distributed ]]
[2026-06-24T21:24:26.314Z] + [[ default == *operator_benchmark* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *operator_microbenchmark* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *attention_microbenchmark* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *inductor_distributed* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *inductor-halide* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *inductor-pallas* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *inductor-triton-cpu* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *inductor-micro-benchmark* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *aoti_cross_compile_for_windows* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *huggingface* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *timm* ]]
[2026-06-24T21:24:26.314Z] + [[ default == cachebench ]]
[2026-06-24T21:24:26.314Z] + [[ default == verify_cachebench ]]
[2026-06-24T21:24:26.314Z] + [[ default == *torchbench* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *inductor_cpp_wrapper* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *inductor_core* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *inductor* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *einops* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *dynamo_core* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *dynamo_cpython* ]]
[2026-06-24T21:24:26.314Z] + [[ default == *dynamo_wrapped* ]]
[2026-06-24T21:24:26.314Z] + [[ pytorch-linux-noble-rocm7.2.4-py3.12-test2 == *rocm* ]]
[2026-06-24T21:24:26.314Z] + [[ -n test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor ]]
[2026-06-24T21:24:26.314Z] + install_torchvision
[2026-06-24T21:24:26.314Z] + local orig_preload
[2026-06-24T21:24:26.314Z] + local commit
[2026-06-24T21:24:26.314Z] ++ get_pinned_commit vision
[2026-06-24T21:24:26.314Z] ++ cat .github/ci_commit_pins/vision.txt
[2026-06-24T21:24:26.314Z] + commit=601776dc14ab12179412fa7fb08762e20862720c
[2026-06-24T21:24:26.314Z] + orig_preload=
[2026-06-24T21:24:26.314Z] + '[' -n '' ']'
[2026-06-24T21:24:26.314Z] + [[ pytorch-linux-noble-rocm7.2.4-py3.12-test2 == *cuda* ]]
[2026-06-24T21:24:26.314Z] + pip_build_and_install git+https://github.com/pytorch/vision.git@601776dc14ab12179412fa7fb08762e20862720c dist/vision
[2026-06-24T21:24:26.314Z] + local build_target=git+https://github.com/pytorch/vision.git@601776dc14ab12179412fa7fb08762e20862720c
[2026-06-24T21:24:26.314Z] + local wheel_dir=dist/vision
[2026-06-24T21:24:26.314Z] + local found_whl=0
[2026-06-24T21:24:26.314Z] + for file in "${wheel_dir}"/*.whl
[2026-06-24T21:24:26.314Z] + [[ -f dist/vision/*.whl ]]
[2026-06-24T21:24:26.314Z] + '[' 0 == 0 ']'
[2026-06-24T21:24:26.314Z] + python3 -m pip wheel --no-build-isolation --no-deps -w dist/vision git+https://github.com/pytorch/vision.git@601776dc14ab12179412fa7fb08762e20862720c
[2026-06-24T21:24:26.728Z] Collecting git+https://github.com/pytorch/vision.git@601776dc14ab12179412fa7fb08762e20862720c
[2026-06-24T21:24:26.728Z] Cloning https://github.com/pytorch/vision.git (to revision 601776dc14ab12179412fa7fb08762e20862720c) to /tmp/pip-req-build-yzsfh0z5
[2026-06-24T21:24:26.728Z] Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-yzsfh0z5
[2026-06-24T21:24:30.918Z] Running command git rev-parse -q --verify 'sha^601776dc14ab12179412fa7fb08762e20862720c'
[2026-06-24T21:24:30.918Z] Running command git fetch -q https://github.com/pytorch/vision.git 601776dc14ab12179412fa7fb08762e20862720c
[2026-06-24T21:24:30.918Z] Running command git checkout -q 601776dc14ab12179412fa7fb08762e20862720c
[2026-06-24T21:24:32.230Z] Resolved https://github.com/pytorch/vision.git to commit 601776dc14ab12179412fa7fb08762e20862720c
[2026-06-24T21:24:32.230Z] Preparing metadata (pyproject.toml): started
[2026-06-24T21:24:34.750Z] Preparing metadata (pyproject.toml): finished with status 'done'
[2026-06-24T21:24:34.750Z] Building wheels for collected packages: torchvision
[2026-06-24T21:24:34.750Z] Building wheel for torchvision (pyproject.toml): started
[2026-06-24T21:25:45.290Z] Building wheel for torchvision (pyproject.toml): still running...
[2026-06-24T21:25:50.600Z] Building wheel for torchvision (pyproject.toml): finished with status 'done'
[2026-06-24T21:25:50.600Z] Created wheel for torchvision: filename=torchvision-0.27.0a0+601776d-cp312-cp312-linux_x86_64.whl size=1568719 sha256=216eecd58be0c924e103e6ff38ba2e0800401630d9244eb138e144d9fc982c9e
[2026-06-24T21:25:50.600Z] Stored in directory: /root/.cache/pip/wheels/b0/fd/40/766dc97b540dba2346c585406c82fefe154a5c1c5b593c8b58
[2026-06-24T21:25:50.600Z] Successfully built torchvision
[2026-06-24T21:25:50.600Z]
[2026-06-24T21:25:50.600Z] [notice] A new release of pip is available: 26.0.1 -> 26.1.2
[2026-06-24T21:25:50.600Z] [notice] To update, run: pip install --upgrade pip
[2026-06-24T21:25:50.600Z] + for file in "${wheel_dir}"/*.whl
[2026-06-24T21:25:50.600Z] + pip_install_whl dist/vision/torchvision-0.27.0a0+601776d-cp312-cp312-linux_x86_64.whl
[2026-06-24T21:25:50.600Z] + args=('dist/vision/torchvision-0.27.0a0+601776d-cp312-cp312-linux_x86_64.whl')
[2026-06-24T21:25:50.600Z] + local args
[2026-06-24T21:25:50.600Z] + [[ dist/vision/torchvision-0.27.0a0+601776d-cp312-cp312-linux_x86_64.whl == *\ * ]]
[2026-06-24T21:25:50.600Z] + for path in "${args[@]}"
[2026-06-24T21:25:50.600Z] + echo 'Installing dist/vision/torchvision-0.27.0a0+601776d-cp312-cp312-linux_x86_64.whl'
[2026-06-24T21:25:50.600Z] + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.27.0a0+601776d-cp312-cp312-linux_x86_64.whl
[2026-06-24T21:25:50.600Z] Installing dist/vision/torchvision-0.27.0a0+601776d-cp312-cp312-linux_x86_64.whl
[2026-06-24T21:25:50.985Z] Processing ./dist/vision/torchvision-0.27.0a0+601776d-cp312-cp312-linux_x86_64.whl
[2026-06-24T21:25:50.985Z] Installing collected packages: torchvision
[2026-06-24T21:25:51.375Z] Successfully installed torchvision-0.27.0a0+601776d
[2026-06-24T21:25:51.375Z] WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
[2026-06-24T21:25:51.375Z] + '[' -n '' ']'
[2026-06-24T21:25:51.375Z] + test_python_shard 2
[2026-06-24T21:25:51.375Z] + [[ -z 2 ]]
[2026-06-24T21:25:51.375Z] + python test/run_test.py --exclude-jit-executor --exclude-distributed-tests --exclude-quantization-tests --include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor --shard 2 2 --verbose --upload-artifacts-while-running
[2026-06-24T21:25:53.267Z] W0624 21:25:53.050000 9623 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-24T21:25:54.577Z] JOB_NAME=None
[2026-06-24T21:25:54.577Z] BUILD_ENVIRONMENT=pytorch-linux-noble-rocm7.2.4-py3.12-test2
[2026-06-24T21:25:54.577Z] test-times lookup key=pytorch-linux-noble-rocm7.2.4-py3.12-test2, test_config=default
[2026-06-24T21:25:54.577Z] ::warning:: Gathered no stats from artifacts for pytorch-linux-noble-rocm7.2.4-py3.12-test2 build env and default test config. Using default job name and default test config instead.
[2026-06-24T21:25:54.577Z] JOB_NAME=None
[2026-06-24T21:25:54.578Z] BUILD_ENVIRONMENT=pytorch-linux-noble-rocm7.2.4-py3.12-test2
[2026-06-24T21:25:54.578Z] test-times lookup key=pytorch-linux-noble-rocm7.2.4-py3.12-test2, test_config=default
[2026-06-24T21:25:54.578Z] ::warning:: Gathered no stats from artifacts for pytorch-linux-noble-rocm7.2.4-py3.12-test2 build env and default test config. Using default job name and default test config instead.
[2026-06-24T21:25:54.578Z] Running all tests
[2026-06-24T21:25:54.578Z] Running parallel tests on 8 processes
[2026-06-24T21:25:54.578Z] Name: tests to run (est. time: 44.71min)
[2026-06-24T21:25:54.578Z] Serial tests (0):
[2026-06-24T21:25:54.578Z] Parallel tests (10):
[2026-06-24T21:25:54.578Z] inductor/test_torchinductor 1/4
[2026-06-24T21:25:54.578Z] inductor/test_torchinductor 2/4
[2026-06-24T21:25:54.578Z] inductor/test_torchinductor 3/4
[2026-06-24T21:25:54.578Z] inductor/test_torchinductor 4/4
[2026-06-24T21:25:54.578Z] test_autograd 1/1
[2026-06-24T21:25:54.578Z] test_binary_ufuncs 1/1
[2026-06-24T21:25:54.578Z] test_ops 2/8
[2026-06-24T21:25:54.578Z] test_ops 3/8
[2026-06-24T21:25:54.578Z] test_ops 6/8
[2026-06-24T21:25:54.578Z] test_ops 7/8
[2026-06-24T21:25:54.578Z] Name: excluded (est. time: 0.0min)
[2026-06-24T21:25:54.578Z] Serial tests (0):
[2026-06-24T21:25:54.578Z] Parallel tests (0):
[2026-06-24T21:25:54.578Z] Running inductor/test_torchinductor 1/4 ... [2026-06-24 21:25:54.300267][200486.913168297]
[2026-06-24T21:25:54.578Z] SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
[2026-06-24T21:25:54.578Z] Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'inductor/test_torchinductor.py', '-m', 'serial', '--shard-id=1', '--num-shards=4', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2026-06-24 21:25:54.300539]
[2026-06-24T21:25:54.578Z] No TD results found
[2026-06-24T21:25:54.578Z] Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=XRBoKk5TT4f6n48PbQ9OntiMMQveSs3J to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
[2026-06-24T21:25:54.578Z] Ignoring disabled issues: ['']
[2026-06-24T21:26:02.492Z]
[2026-06-24T21:26:02.492Z] PRINTING LOG FILE of inductor/test_torchinductor 1/4 (test/test-reports/inductor.test_torchinductor_1.4_ce2f311899feb40a_.log)
[2026-06-24T21:26:02.492Z] W0624 21:25:57.520000 9697 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-24T21:26:02.492Z] Traceback (most recent call last):
[2026-06-24T21:26:02.492Z] File "/var/lib/jenkins/pytorch/test/inductor/test_torchinductor.py", line 1099, in <module>
[2026-06-24T21:26:02.492Z] class CommonTemplate:
[2026-06-24T21:26:02.492Z] File "/var/lib/jenkins/pytorch/test/inductor/test_torchinductor.py", line 6579, in CommonTemplate
[2026-06-24T21:26:02.492Z] @skipIfMPS
[2026-06-24T21:26:02.492Z] ^^^^^^^^^
[2026-06-24T21:26:02.492Z] NameError: name 'skipIfMPS' is not defined. Did you mean: 'skipIfXpu'?
[2026-06-24T21:26:02.492Z] Got exit code 1
[2026-06-24T21:26:02.492Z] No stepcurrent file found. Either pytest didn't get to run (e.g. import error) or file got deleted (contact dev infra)
[2026-06-24T21:26:02.492Z]
[2026-06-24T21:26:02.492Z] FINISHED PRINTING LOG FILE of inductor/test_torchinductor 1/4 (test/test-reports/inductor.test_torchinductor_1.4_ce2f311899feb40a_.log)
[2026-06-24T21:26:02.492Z]
[2026-06-24T21:26:02.492Z] Finished inductor/test_torchinductor 1/4 ... [2026-06-24 21:26:02.248662][200494.861564556], took 0.13min
[2026-06-24T21:26:02.492Z] inductor/test_torchinductor 1/4 failed!
[2026-06-24T21:26:02.492Z] Emitting td_test_failure_stats_v2
[2026-06-24T21:26:02.492Z] /var/lib/jenkins/pytorch/tools/stats/upload_metrics.py:140: UserWarning: Not emitting metrics for td_test_failure_stats_v2. Missing repo. Please set the GITHUB_REPOSITORY environment variable to pass in this value.
[2026-06-24T21:26:02.492Z] warn(f"Not emitting metrics for {metric_name}. {e}")
[2026-06-24T21:26:02.492Z] GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
[2026-06-24T21:26:02.492Z] Uploading artifacts took 0.00 seconds
[2026-06-24T21:26:02.492Z] Traceback (most recent call last):
[2026-06-24T21:26:02.492Z] File "/var/lib/jenkins/pytorch/test/run_test.py", line 2319, in <module>
[2026-06-24T21:26:02.492Z] main()
[2026-06-24T21:26:02.492Z] File "/var/lib/jenkins/pytorch/test/run_test.py", line 2270, in main
[2026-06-24T21:26:02.492Z] run_tests(
[2026-06-24T21:26:02.492Z] File "/var/lib/jenkins/pytorch/test/run_test.py", line 2106, in run_tests
[2026-06-24T21:26:02.492Z] raise RuntimeError(failure.message + keep_going_message)
[2026-06-24T21:26:02.492Z] RuntimeError: inductor/test_torchinductor 1/4 failed!
[2026-06-24T21:26:02.492Z]
[2026-06-24T21:26:02.492Z] Tip: You can keep running tests even on failure by passing --keep-going to run_test.py.
[2026-06-24T21:26:02.492Z] If running on CI, add the 'keep-going' label to your PR and rerun your jobs.
[2026-06-24T21:26:02.880Z]
[2026-06-24T21:26:02.880Z] real 0m11.577s
[2026-06-24T21:26:02.880Z] user 0m26.176s
[2026-06-24T21:26:02.880Z] sys 0m1.650s
[2026-06-24T21:26:02.880Z] + sccache_epilogue
[2026-06-24T21:26:02.880Z] + echo '::group::Sccache Compilation Log'
[2026-06-24T21:26:02.880Z] + echo '=================== sccache compilation log ==================='
[2026-06-24T21:26:02.880Z] ::group::Sccache Compilation Log
[2026-06-24T21:26:02.880Z] =================== sccache compilation log ===================
[2026-06-24T21:26:02.880Z] + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /root/sccache_error.log
[2026-06-24T21:26:02.880Z] + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
[2026-06-24T21:26:02.880Z] + sccache --show-stats
[2026-06-24T21:26:02.880Z] =========== If your build fails, please take a look at the log above for possible reasons ===========
[2026-06-24T21:26:02.880Z] Compile requests 60
[2026-06-24T21:26:02.880Z] Compile requests executed 41
[2026-06-24T21:26:02.880Z] Cache hits 0
[2026-06-24T21:26:02.880Z] Cache misses 41
[2026-06-24T21:26:02.880Z] Cache misses (C/C++) 34
[2026-06-24T21:26:02.880Z] Cache misses (HIP) 7
[2026-06-24T21:26:02.880Z] Cache hits rate 0.00 %
[2026-06-24T21:26:02.880Z] Cache hits rate (C/C++) 0.00 %
[2026-06-24T21:26:02.880Z] Cache hits rate (HIP) 0.00 %
[2026-06-24T21:26:02.880Z] Cache timeouts 0
[2026-06-24T21:26:02.880Z] Cache read errors 0
[2026-06-24T21:26:02.880Z] Forced recaches 0
[2026-06-24T21:26:02.880Z] Cache write errors 0
[2026-06-24T21:26:02.880Z] Cache errors 0
[2026-06-24T21:26:02.880Z] Compilations 41
[2026-06-24T21:26:02.880Z] Compilation failures 0
[2026-06-24T21:26:02.880Z] Non-cacheable compilations 0
[2026-06-24T21:26:02.880Z] Non-cacheable calls 0
[2026-06-24T21:26:02.880Z] Non-compilation calls 19
[2026-06-24T21:26:02.880Z] Unsupported compiler calls 0
[2026-06-24T21:26:02.880Z] Average cache write 0.000 s
[2026-06-24T21:26:02.880Z] Average compiler 15.063 s
[2026-06-24T21:26:02.880Z] Average cache read hit 0.000 s
[2026-06-24T21:26:02.880Z] Failed distributed compilations 0
[2026-06-24T21:26:02.880Z] Cache location Local disk: "/root/.cache/sccache"
[2026-06-24T21:26:02.880Z] Use direct/preprocessor mode? yes
[2026-06-24T21:26:02.880Z] Version (client) 0.13.0
[2026-06-24T21:26:02.880Z] Cache size 3 MiB
[2026-06-24T21:26:02.880Z] Max cache size 10 GiB
[2026-06-24T21:26:02.880Z] + sccache --stop-server
[2026-06-24T21:26:02.880Z] Stopping sccache server...
[2026-06-24T21:26:02.880Z] Compile requests 60
[2026-06-24T21:26:02.880Z] Compile requests executed 41
[2026-06-24T21:26:02.880Z] Cache hits 0
[2026-06-24T21:26:02.880Z] Cache misses 41
[2026-06-24T21:26:02.880Z] Cache misses (C/C++) 34
[2026-06-24T21:26:02.880Z] Cache misses (HIP) 7
[2026-06-24T21:26:02.880Z] Cache hits rate 0.00 %
[2026-06-24T21:26:02.880Z] Cache hits rate (C/C++) 0.00 %
[2026-06-24T21:26:02.880Z] Cache hits rate (HIP) 0.00 %
[2026-06-24T21:26:02.880Z] Cache timeouts 0
[2026-06-24T21:26:02.880Z] Cache read errors 0
[2026-06-24T21:26:02.880Z] Forced recaches 0
[2026-06-24T21:26:02.880Z] Cache write errors 0
[2026-06-24T21:26:02.880Z] Cache errors 0
[2026-06-24T21:26:02.880Z] Compilations 41
[2026-06-24T21:26:02.880Z] Compilation failures 0
[2026-06-24T21:26:02.880Z] Non-cacheable compilations 0
[2026-06-24T21:26:02.880Z] Non-cacheable calls 0
[2026-06-24T21:26:02.880Z] Non-compilation calls 19
[2026-06-24T21:26:02.880Z] Unsupported compiler calls 0
[2026-06-24T21:26:02.880Z] Average cache write 0.000 s
[2026-06-24T21:26:02.880Z] Average compiler 15.063 s
[2026-06-24T21:26:02.880Z] Average cache read hit 0.000 s
[2026-06-24T21:26:02.880Z] Failed distributed compilations 0
[2026-06-24T21:26:02.880Z] Cache location Local disk: "/root/.cache/sccache"
[2026-06-24T21:26:02.880Z] Use direct/preprocessor mode? yes
[2026-06-24T21:26:02.880Z] Version (client) 0.13.0
[2026-06-24T21:26:02.880Z] Cache size 3 MiB
[2026-06-24T21:26:02.880Z] Max cache size 10 GiB
[2026-06-24T21:26:02.880Z] ::endgroup::
[2026-06-24T21:26:02.880Z] + echo ::endgroup::
[2026-06-24T21:26:02.880Z] + cp -RT test/test-reports /host_workspace/pytorch_reports
[2026-06-24T21:26:02.880Z] + chmod -R 777 /host_workspace/pytorch_log /host_workspace/pytorch_reports
[2026-06-24T21:26:02.880Z] + git clean -fdx
[2026-06-24T21:26:03.300Z] Removing .additional_ci_files/
[2026-06-24T21:26:03.300Z] Removing .pytest_cache/
[2026-06-24T21:26:03.300Z] Removing build/
[2026-06-24T21:26:03.300Z] Removing dist/
[2026-06-24T21:26:03.300Z] Removing test/.pytorch-disabled-tests.json
[2026-06-24T21:26:03.300Z] Removing test/test-reports/
[2026-06-24T21:26:03.300Z] Removing test_artifacts.zip
[2026-06-24T21:26:03.300Z] Removing tools/__pycache__/
[2026-06-24T21:26:03.300Z] Removing tools/stats/__pycache__/
[2026-06-24T21:26:03.300Z] Removing tools/testing/__pycache__/
[2026-06-24T21:26:03.300Z] Removing tools/testing/target_determination/__pycache__/
[2026-06-24T21:26:03.300Z] Removing tools/testing/target_determination/heuristics/__pycache__/
[2026-06-24T21:26:03.300Z] Removing torch-2.12.0+git0173828-cp312-cp312-linux_x86_64.whl
Output truncated.
Details
- Kill older PR Builds (2.8 sec)
- Initialize (1 hr 5 min)
- Download CI scripts (26 sec)
- Checkout Pytorch (1 min 24 sec)
- Check base Docker image existence (8.8 sec)
- Pull Docker Image (6 min 7 sec)
- Build PyTorch (56 min)
- Tests (11 hr)
- Test PyTorch (8 ms)
- Test Distributed (9 ms)
- Test Inductor (7 ms)
- Test PyTorch Slow (7 ms)
- Test PyTorch Slow (13 sec)
- Microbenchmark (27 sec)
- Microbenchmark (13 sec)
- Post Build (2.3 sec)
- Declarative: Post Actions (4.3 sec)
Loading