[release/2.12] - Update triton commit and update logic to build from branch instead of triton.txt#3266
Open
jataylo wants to merge 2 commits into
Open
[release/2.12] - Update triton commit and update logic to build from branch instead of triton.txt#3266jataylo wants to merge 2 commits into
jataylo wants to merge 2 commits into
ROCm Repo Management API / Jenkins
failed
Jun 2, 2026 in 9h 41m 5s
Tests/Test Distributed/Run pytorch_distributed_2: warning in 'junit' step
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
Build log truncated.
[2026-06-02T20:29:41.617Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.617Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.617Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.617Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.617Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.617Z] configfile: pytest.ini
[2026-06-02T20:29:41.617Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.617Z] collecting ... collected 1 item
[2026-06-02T20:29:41.617Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.617Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0005s] (Nccl does not support CPU tensors) [100%]
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fdee0f444e8f7d73.xml -
[2026-06-02T20:29:41.617Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.617Z] W0602 20:27:21.611000 2236120 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd53b06202e9705f.xml
[2026-06-02T20:29:41.617Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.617Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.617Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.617Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.617Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.617Z] configfile: pytest.ini
[2026-06-02T20:29:41.617Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.617Z] collecting ... collected 1 item
[2026-06-02T20:29:41.617Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.617Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py I0602 20:27:22.816000 2236120 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 0 with pid 2236193
[2026-06-02T20:29:41.617Z] I0602 20:27:22.817000 2236120 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 1 with pid 2236194
[2026-06-02T20:29:41.617Z] I0602 20:27:22.818000 2236120 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 2 with pid 2236195
[2026-06-02T20:29:41.617Z] I0602 20:27:22.819000 2236120 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 3 with pid 2236196
[2026-06-02T20:29:41.617Z] W0602 20:27:24.770000 2236195 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] W0602 20:27:24.785000 2236193 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] W0602 20:27:24.839000 2236194 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] W0602 20:27:24.846000 2236196 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] [rank2]:W0602 20:27:25.899000 2236195 site-packages/torch/distributed/distributed_c10d.py:3344] _object_to_tensor size: 511 hash value: 12668507973458366732
[2026-06-02T20:29:41.617Z] [rank3]:W0602 20:27:25.902000 2236196 site-packages/torch/distributed/distributed_c10d.py:3344] _object_to_tensor size: 18 hash value: 6002723302436841527
[2026-06-02T20:29:41.617Z] [rank0]:W0602 20:27:25.947000 2236193 site-packages/torch/distributed/distributed_c10d.py:3344] _object_to_tensor size: 54 hash value: 15559783960315060411
[2026-06-02T20:29:41.617Z] [rank1]:W0602 20:27:25.988000 2236194 site-packages/torch/distributed/distributed_c10d.py:3344] _object_to_tensor size: 97 hash value: 2395169942717765378
[2026-06-02T20:29:41.617Z] [rank0]:W0602 20:27:30.953000 2236193 site-packages/torch/distributed/distributed_c10d.py:3359] _tensor_to_object size: 511 hash value: 16444151079320822176
[2026-06-02T20:29:41.617Z] [rank0]:W0602 20:27:30.954000 2236193 site-packages/torch/distributed/distributed_c10d.py:3359] _tensor_to_object size: 511 hash value: 16444151079320822176
[2026-06-02T20:29:41.617Z] [rank0]:W0602 20:27:30.954000 2236193 site-packages/torch/distributed/distributed_c10d.py:3359] _tensor_to_object size: 511 hash value: 16444151079320822176
[2026-06-02T20:29:41.617Z] [rank0]:W0602 20:27:30.956000 2236193 site-packages/torch/distributed/distributed_c10d.py:3359] _tensor_to_object size: 511 hash value: 16444151079320822176
[2026-06-02T20:29:41.617Z] /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
[2026-06-02T20:29:41.617Z] return func(*args, **kwargs)
[2026-06-02T20:29:41.617Z] PASSED [9.6417s] [100%]
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd53b06202e9705f.xml -
[2026-06-02T20:29:41.617Z] ============================== 1 passed in 9.66s ===============================
[2026-06-02T20:29:41.617Z] W0602 20:27:34.970000 2236539 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f597623b757f88c2.xml
[2026-06-02T20:29:41.617Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.617Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.617Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.617Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.617Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.617Z] configfile: pytest.ini
[2026-06-02T20:29:41.617Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.617Z] collecting ... collected 1 item
[2026-06-02T20:29:41.617Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.617Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0005s] (Test requires backend nccl to be one of {'gloo'}) [100%]
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f597623b757f88c2.xml -
[2026-06-02T20:29:41.617Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.617Z] W0602 20:27:38.587000 2236611 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36c472d39c8fdf24.xml
[2026-06-02T20:29:41.617Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.617Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.617Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.617Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.617Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.617Z] configfile: pytest.ini
[2026-06-02T20:29:41.617Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.617Z] collecting ... collected 1 item
[2026-06-02T20:29:41.617Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.617Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py I0602 20:27:39.805000 2236611 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 0 with pid 2236684
[2026-06-02T20:29:41.617Z] I0602 20:27:39.806000 2236611 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 1 with pid 2236685
[2026-06-02T20:29:41.617Z] I0602 20:27:39.807000 2236611 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 2 with pid 2236686
[2026-06-02T20:29:41.617Z] I0602 20:27:39.808000 2236611 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 3 with pid 2236687
[2026-06-02T20:29:41.617Z] W0602 20:27:41.690000 2236686 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] W0602 20:27:41.788000 2236684 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] W0602 20:27:41.817000 2236685 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] W0602 20:27:41.825000 2236687 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] PASSED [4.0352s] [100%]
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-36c472d39c8fdf24.xml -
[2026-06-02T20:29:41.617Z] ============================== 1 passed in 4.06s ===============================
[2026-06-02T20:29:41.617Z] W0602 20:27:46.408000 2237033 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-439b3f51cdc3954f.xml
[2026-06-02T20:29:41.617Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.617Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.617Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.617Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.617Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.617Z] configfile: pytest.ini
[2026-06-02T20:29:41.617Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.617Z] collecting ... collected 1 item
[2026-06-02T20:29:41.617Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.617Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py I0602 20:27:47.623000 2237033 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 0 with pid 2237106
[2026-06-02T20:29:41.617Z] I0602 20:27:47.625000 2237033 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 1 with pid 2237107
[2026-06-02T20:29:41.617Z] I0602 20:27:47.625000 2237033 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 2 with pid 2237108
[2026-06-02T20:29:41.617Z] I0602 20:27:47.626000 2237033 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 3 with pid 2237109
[2026-06-02T20:29:41.617Z] W0602 20:27:49.624000 2237108 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] W0602 20:27:49.627000 2237106 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] W0602 20:27:49.663000 2237109 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] W0602 20:27:49.664000 2237107 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
[2026-06-02T20:29:41.617Z] return func(*args, **kwargs)
[2026-06-02T20:29:41.617Z] PASSED [10.7437s] [100%]
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-439b3f51cdc3954f.xml -
[2026-06-02T20:29:41.617Z] ============================== 1 passed in 10.77s ==============================
[2026-06-02T20:29:41.617Z] W0602 20:28:00.869000 2237462 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.617Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d08aee071f4b167.xml
[2026-06-02T20:29:41.617Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.617Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.617Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.617Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.617Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.617Z] configfile: pytest.ini
[2026-06-02T20:29:41.617Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.617Z] collecting ... collected 1 item
[2026-06-02T20:29:41.617Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.617Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view
[2026-06-02T20:29:41.617Z]
[2026-06-02T20:29:41.617Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py I0602 20:28:02.085000 2237462 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 0 with pid 2237535
[2026-06-02T20:29:41.617Z] I0602 20:28:02.086000 2237462 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 1 with pid 2237536
[2026-06-02T20:29:41.617Z] I0602 20:28:02.087000 2237462 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 2 with pid 2237537
[2026-06-02T20:29:41.618Z] I0602 20:28:02.088000 2237462 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 3 with pid 2237538
[2026-06-02T20:29:41.618Z] W0602 20:28:03.984000 2237535 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:28:04.059000 2237537 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:28:04.123000 2237538 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:28:04.152000 2237536 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] PASSED [11.6478s] [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d08aee071f4b167.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 passed in 11.67s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:28:16.073000 2238376 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c0c1c48d70d5b30d.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0005s] (Nccl does not support CPU tensors) [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c0c1c48d70d5b30d.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:28:19.650000 2238505 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f00ca772f6f7ddd9.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0005s] (Nccl does not support CPU tensors) [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f00ca772f6f7ddd9.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:28:23.225000 2238577 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c816453f6443b9d2.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0005s] (Nccl does not support CPU tensors) [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c816453f6443b9d2.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:28:26.733000 2238649 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20eb3f72273c6cf4.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0005s] (Nccl does not support CPU tensors) [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20eb3f72273c6cf4.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:28:30.304000 2238721 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fe8c443d493e8b2.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0005s] (Nccl does not support CPU tensors) [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2fe8c443d493e8b2.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:28:33.807000 2238793 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e4bd8062ae64c69.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0005s] (Nccl send/recv tested by test_send_recv_nccl) [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e4bd8062ae64c69.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:28:37.284000 2238867 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b25a5574ab0e7c56.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0004s] (nccl does not support send/recv from any source) [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b25a5574ab0e7c56.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:28:40.807000 2238939 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c06508c4bf63202.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py I0602 20:28:42.021000 2238939 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 0 with pid 2239012
[2026-06-02T20:29:41.618Z] I0602 20:28:42.022000 2238939 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 1 with pid 2239013
[2026-06-02T20:29:41.618Z] I0602 20:28:42.022000 2238939 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 2 with pid 2239014
[2026-06-02T20:29:41.618Z] I0602 20:28:42.023000 2238939 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 3 with pid 2239015
[2026-06-02T20:29:41.618Z] W0602 20:28:43.962000 2239015 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:28:44.009000 2239014 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:28:44.053000 2239013 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:28:44.059000 2239012 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/profiler/profiler.py:272: UserWarning: Warning: Profiler clears events at the end of each cycle.Only events from the current cycle will be reported.To keep events across cycles, set acc_events=True.
[2026-06-02T20:29:41.618Z] _warn_once(
[2026-06-02T20:29:41.618Z] USDT:2026-06-02 20:28:45 2239015:2239015 ActivityProfilerController.cpp:415] profiler_start
[2026-06-02T20:29:41.618Z] [rank3]:[W602 20:28:45.535673413 ProcessGroupNCCL.cpp:4215] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 4. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator())
[2026-06-02T20:29:41.618Z] /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/profiler/profiler.py:272: UserWarning: Warning: Profiler clears events at the end of each cycle.Only events from the current cycle will be reported.To keep events across cycles, set acc_events=True.
[2026-06-02T20:29:41.618Z] _warn_once(
[2026-06-02T20:29:41.618Z] USDT:2026-06-02 20:28:45 2239014:2239014 ActivityProfilerController.cpp:415] profiler_start
[2026-06-02T20:29:41.618Z] [rank2]:[W602 20:28:45.555107988 ProcessGroupNCCL.cpp:4215] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 4. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator())
[2026-06-02T20:29:41.618Z] /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/profiler/profiler.py:272: UserWarning: Warning: Profiler clears events at the end of each cycle.Only events from the current cycle will be reported.To keep events across cycles, set acc_events=True.
[2026-06-02T20:29:41.618Z] _warn_once(
[2026-06-02T20:29:41.618Z] USDT:2026-06-02 20:28:45 2239012:2239012 ActivityProfilerController.cpp:415] profiler_start
[2026-06-02T20:29:41.618Z] [rank0]:[W602 20:28:45.596170093 ProcessGroupNCCL.cpp:4215] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 4. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator())
[2026-06-02T20:29:41.618Z] /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/profiler/profiler.py:272: UserWarning: Warning: Profiler clears events at the end of each cycle.Only events from the current cycle will be reported.To keep events across cycles, set acc_events=True.
[2026-06-02T20:29:41.618Z] _warn_once(
[2026-06-02T20:29:41.618Z] USDT:2026-06-02 20:28:45 2239013:2239013 ActivityProfilerController.cpp:415] profiler_start
[2026-06-02T20:29:41.618Z] [rank1]:[W602 20:28:45.618108644 ProcessGroupNCCL.cpp:4215] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 4. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator())
[2026-06-02T20:29:41.618Z] [rank1]:[W602 20:28:49.881732582 ProcessGroupNCCL.cpp:4215] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 4. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator())
[2026-06-02T20:29:41.618Z] [rank0]:[W602 20:28:57.005967586 ProcessGroupNCCL.cpp:4215] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 4. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator())
[2026-06-02T20:29:41.618Z] [rank2]:[W602 20:28:57.256823798 ProcessGroupNCCL.cpp:4215] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 4. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator())
[2026-06-02T20:29:41.618Z] [rank3]:[W602 20:28:58.877551885 ProcessGroupNCCL.cpp:4215] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 4. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator())
[2026-06-02T20:29:41.618Z] USDT:2026-06-02 20:28:58 2239014:2239014 ActivityProfilerController.cpp:455] profiler_stop
[2026-06-02T20:29:41.618Z] [rank2]:[W602 20:28:58.903135378 collection.cpp:1182] Warning: ROCTracer produced duplicate flow start: 4 (function operator())
[2026-06-02T20:29:41.618Z] USDT:2026-06-02 20:28:58 2239012:2239012 ActivityProfilerController.cpp:455] profiler_stop
[2026-06-02T20:29:41.618Z] USDT:2026-06-02 20:28:58 2239013:2239013 ActivityProfilerController.cpp:455] profiler_stop
[2026-06-02T20:29:41.618Z] [rank0]:[W602 20:28:58.988814891 collection.cpp:1182] Warning: ROCTracer produced duplicate flow start: 3 (function operator())
[2026-06-02T20:29:41.618Z] [rank1]:[W602 20:28:58.992772857 collection.cpp:1182] Warning: ROCTracer produced duplicate flow start: 4 (function operator())
[2026-06-02T20:29:41.618Z] USDT:2026-06-02 20:28:58 2239015:2239015 ActivityProfilerController.cpp:455] profiler_stop
[2026-06-02T20:29:41.618Z] [rank3]:[W602 20:28:58.002174579 collection.cpp:1182] Warning: ROCTracer produced duplicate flow start: 4 (function operator())
[2026-06-02T20:29:41.618Z] PASSED [18.5509s] [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7c06508c4bf63202.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 passed in 18.58s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:29:02.909000 2239492 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f59f9d5b7567185d.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py SKIPPED [0.0005s] (Only Gloo backend support sparse all reduce) [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f59f9d5b7567185d.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 skipped in 0.02s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:29:06.432000 2239564 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b87a34b81222705.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py I0602 20:29:07.633000 2239564 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 0 with pid 2239760
[2026-06-02T20:29:41.618Z] I0602 20:29:07.634000 2239564 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 1 with pid 2239761
[2026-06-02T20:29:41.618Z] I0602 20:29:07.634000 2239564 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 2 with pid 2239762
[2026-06-02T20:29:41.618Z] I0602 20:29:07.635000 2239564 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 3 with pid 2239763
[2026-06-02T20:29:41.618Z] W0602 20:29:09.512000 2239761 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:29:09.673000 2239763 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:29:09.690000 2239762 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:29:09.715000 2239760 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] PASSED [10.3483s] [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6b87a34b81222705.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 passed in 10.37s ==============================
[2026-06-02T20:29:41.618Z] W0602 20:29:20.333000 2240470 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] Test results will be stored in test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ddef60cb626cc739.xml
[2026-06-02T20:29:41.618Z] ============================= test session starts ==============================
[2026-06-02T20:29:41.618Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T20:29:41.618Z] cachedir: .pytest_cache
[2026-06-02T20:29:41.618Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T20:29:41.618Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T20:29:41.618Z] configfile: pytest.ini
[2026-06-02T20:29:41.618Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-06-02T20:29:41.618Z] collecting ... collected 1 item
[2026-06-02T20:29:41.618Z] stepcurrent: previously run test not found, not skipping.
[2026-06-02T20:29:41.618Z] Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py I0602 20:29:21.553000 2240470 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 0 with pid 2240543
[2026-06-02T20:29:41.618Z] I0602 20:29:21.554000 2240470 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 1 with pid 2240544
[2026-06-02T20:29:41.618Z] I0602 20:29:21.554000 2240470 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 2 with pid 2240545
[2026-06-02T20:29:41.618Z] I0602 20:29:21.555000 2240470 site-packages/torch/testing/_internal/common_distributed.py:895] Started process 3 with pid 2240546
[2026-06-02T20:29:41.618Z] W0602 20:29:23.547000 2240545 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:29:23.551000 2240543 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:29:23.607000 2240546 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] W0602 20:29:23.622000 2240544 site-packages/torch/_native/cutedsl_utils.py:55] CuTeDSL operators require optional Python packages `nvidia-cutlass-dsl` and `apache-tvm-ffi`; missing optional dependency `nvidia_cutlass_dsl` (importlib.util.find_spec(nvidia_cutlass_dsl) failed)
[2026-06-02T20:29:41.618Z] PASSED [9.6430s] [100%]
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ddef60cb626cc739.xml -
[2026-06-02T20:29:41.618Z] ============================== 1 passed in 9.67s ===============================
[2026-06-02T20:29:41.618Z] Traceback (most recent call last):
[2026-06-02T20:29:41.618Z] File "/var/lib/jenkins/pytorch/test/distributed/test_distributed_spawn.py", line 60, in <module>
[2026-06-02T20:29:41.618Z] run_tests()
[2026-06-02T20:29:41.618Z] File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1327, in run_tests
[2026-06-02T20:29:41.618Z] raise AssertionError(
[2026-06-02T20:29:41.618Z] AssertionError: 1 unit test(s) failed:
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn 5/7 (test/test-reports/distributed.test_distributed_spawn_5.7_0e1870279d6eb334_.log)
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] Finished distributed/test_distributed_spawn 5/7 ... [2026-06-02 20:29:32.575823][32544.997380214], took 18.76min
[2026-06-02T20:29:41.618Z] distributed/test_distributed_spawn 5/7 failed!
[2026-06-02T20:29:41.618Z] Emitting td_test_failure_stats_v2
[2026-06-02T20:29:41.618Z] /var/lib/jenkins/pytorch/tools/stats/upload_metrics.py:140: UserWarning: Not emitting metrics for td_test_failure_stats_v2. Missing repo. Please set the GITHUB_REPOSITORY environment variable to pass in this value.
[2026-06-02T20:29:41.618Z] warn(f"Not emitting metrics for {metric_name}. {e}")
[2026-06-02T20:29:41.618Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f835620d7b22c5ce.xml
[2026-06-02T20:29:41.618Z] Found job id: None
[2026-06-02T20:29:41.618Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-06-02T20:29:41.618Z] GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
[2026-06-02T20:29:41.618Z] Uploading artifacts took 0.00 seconds
[2026-06-02T20:29:41.618Z] Traceback (most recent call last):
[2026-06-02T20:29:41.618Z] File "/var/lib/jenkins/pytorch/test/run_test.py", line 2319, in <module>
[2026-06-02T20:29:41.618Z] main()
[2026-06-02T20:29:41.618Z] File "/var/lib/jenkins/pytorch/test/run_test.py", line 2270, in main
[2026-06-02T20:29:41.618Z] run_tests(
[2026-06-02T20:29:41.618Z] File "/var/lib/jenkins/pytorch/test/run_test.py", line 2091, in run_tests
[2026-06-02T20:29:41.618Z] raise RuntimeError(failure.message + keep_going_message)
[2026-06-02T20:29:41.618Z] RuntimeError: distributed/test_distributed_spawn 5/7 failed!
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] Tip: You can keep running tests even on failure by passing --keep-going to run_test.py.
[2026-06-02T20:29:41.618Z] If running on CI, add the 'keep-going' label to your PR and rerun your jobs.
[2026-06-02T20:29:41.618Z]
[2026-06-02T20:29:41.618Z] real 86m7.868s
[2026-06-02T20:29:41.618Z] user 492m23.079s
[2026-06-02T20:29:41.618Z] sys 71m4.287s
[2026-06-02T20:29:41.618Z] + sccache_epilogue
[2026-06-02T20:29:41.618Z] + echo '::group::Sccache Compilation Log'
[2026-06-02T20:29:41.618Z] + echo '=================== sccache compilation log ==================='
[2026-06-02T20:29:41.618Z] ::group::Sccache Compilation Log
[2026-06-02T20:29:41.618Z] =================== sccache compilation log ===================
[2026-06-02T20:29:41.618Z] + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /root/sccache_error.log
[2026-06-02T20:29:41.618Z] + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
[2026-06-02T20:29:41.618Z] + sccache --show-stats
[2026-06-02T20:29:41.618Z] =========== If your build fails, please take a look at the log above for possible reasons ===========
[2026-06-02T20:29:41.618Z] Compile requests 2
[2026-06-02T20:29:41.618Z] Compile requests executed 0
[2026-06-02T20:29:41.618Z] Cache hits 0
[2026-06-02T20:29:41.618Z] Cache misses 0
[2026-06-02T20:29:41.618Z] Cache hits rate -
[2026-06-02T20:29:41.618Z] Cache timeouts 0
[2026-06-02T20:29:41.618Z] Cache read errors 0
[2026-06-02T20:29:41.618Z] Forced recaches 0
[2026-06-02T20:29:41.618Z] Cache write errors 0
[2026-06-02T20:29:41.618Z] Cache errors 0
[2026-06-02T20:29:41.618Z] Compilations 0
[2026-06-02T20:29:41.618Z] Compilation failures 0
[2026-06-02T20:29:41.618Z] Non-cacheable compilations 0
[2026-06-02T20:29:41.618Z] Non-cacheable calls 0
[2026-06-02T20:29:41.618Z] Non-compilation calls 2
[2026-06-02T20:29:41.618Z] Unsupported compiler calls 0
[2026-06-02T20:29:41.618Z] Average cache write 0.000 s
[2026-06-02T20:29:41.618Z] Average compiler 0.000 s
[2026-06-02T20:29:41.618Z] Average cache read hit 0.000 s
[2026-06-02T20:29:41.618Z] Failed distributed compilations 0
[2026-06-02T20:29:41.618Z] Cache location Local disk: "/root/.cache/sccache"
[2026-06-02T20:29:41.618Z] Use direct/preprocessor mode? yes
[2026-06-02T20:29:41.618Z] Version (client) 0.13.0
[2026-06-02T20:29:41.618Z] Max cache size 10 GiB
[2026-06-02T20:29:41.618Z] + sccache --stop-server
[2026-06-02T20:29:41.618Z] Stopping sccache server...
[2026-06-02T20:29:41.618Z] Compile requests 2
[2026-06-02T20:29:41.618Z] Compile requests executed 0
[2026-06-02T20:29:41.618Z] Cache hits 0
[2026-06-02T20:29:41.618Z] Cache misses 0
[2026-06-02T20:29:41.618Z] Cache hits rate -
[2026-06-02T20:29:41.618Z] Cache timeouts 0
[2026-06-02T20:29:41.618Z] Cache read errors 0
[2026-06-02T20:29:41.618Z] Forced recaches 0
[2026-06-02T20:29:41.618Z] Cache write errors 0
[2026-06-02T20:29:41.618Z] Cache errors 0
[2026-06-02T20:29:41.618Z] Compilations 0
[2026-06-02T20:29:41.618Z] Compilation failures 0
[2026-06-02T20:29:41.618Z] Non-cacheable compilations 0
[2026-06-02T20:29:41.618Z] Non-cacheable calls 0
[2026-06-02T20:29:41.618Z] Non-compilation calls 2
[2026-06-02T20:29:41.618Z] Unsupported compiler calls 0
[2026-06-02T20:29:41.618Z] Average cache write 0.000 s
[2026-06-02T20:29:41.618Z] Average compiler 0.000 s
[2026-06-02T20:29:41.618Z] Average cache read hit 0.000 s
[2026-06-02T20:29:41.618Z] Failed distributed compilations 0
[2026-06-02T20:29:41.618Z] Cache location Local disk: "/root/.cache/sccache"
[2026-06-02T20:29:41.618Z] Use direct/preprocessor mode? yes
[2026-06-02T20:29:41.618Z] Version (client) 0.13.0
[2026-06-02T20:29:41.618Z] Max cache size 10 GiB
[2026-06-02T20:29:41.618Z] + echo ::endgroup::
[2026-06-02T20:29:41.618Z] ::endgroup::
[2026-06-02T20:29:41.618Z] + cp -RT test/test-reports /host_workspace/pytorch_reports
[2026-06-02T20:29:41.618Z] + chmod -R 777 /host_workspace/pytorch_log /host_workspace/pytorch_reports
[2026-06-02T20:29:41.618Z] + git clean -fdx
[2026-06-02T20:29:41.618Z] Removing .additional_ci_files/
[2026-06-02T20:29:41.618Z] Removing .pytest_cache/
[2026-06-02T20:29:41.618Z] Removing build/
[2026-06-02T20:29:41.618Z] Removing dist/
[2026-06-02T20:29:41.618Z] Removing test/.pytorch-disabled-tests.json
[2026-06-02T20:29:41.618Z] Removing test/__pycache__/
[2026-06-02T20:29:41.618Z] Removing test/distributed/__pycache__/
[2026-06-02T20:29:41.618Z] Removing test/test-reports/
[2026-06-02T20:29:41.618Z] Removing test_artifacts.zip
[2026-06-02T20:29:41.618Z] Removing tools/__pycache__/
[2026-06-02T20:29:41.618Z] Removing tools/stats/__pycache__/
[2026-06-02T20:29:41.618Z] Removing tools/testing/__pycache__/
[2026-06-02T20:29:41.618Z] Removing tools/testing/target_determination/__pycache__/
[2026-06-02T20:29:41.618Z] Removing tools/testing/target_determination/heuristics/__pycache__/
[2026-06-02T20:29:41.618Z] Removing torch-2.12.0+git1232565-cp312-cp312-linux_x86_64.whl
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Error signal
Output truncated.
Details
- Kill older PR Builds (1.6 sec)
- Initialize (2 hr 48 min)
- Download CI scripts (31 sec)
- Checkout Pytorch (1 min 21 sec)
- Check base Docker image existence (7.5 sec)
- Build Docker image (1 hr 17 min)
- Build PyTorch (1 hr 26 min)
- Tests (6 hr 52 min)
- Test PyTorch (17 ms)
- Test PyTorch (3 hr 22 min)
- Run pytorch_test_1 (1 hr 44 min)
- Run pytorch_test_2 (1 hr 37 min)
- Test PyTorch (3 hr 22 min)
- Test Distributed (7 ms)
- Test Inductor (10 ms)
- Test Inductor (3 hr 24 min)
- Run pytorch_inductor_1 (3 hr 24 min)
- Test Inductor (3 hr 24 min)
- Test PyTorch Slow (7 ms)
- Test PyTorch Slow (14 sec)
- Microbenchmark (29 sec)
- Microbenchmark (14 sec)
- Test PyTorch (17 ms)
- Post Build (1.8 sec)
- Declarative: Post Actions (4.3 sec)
Loading