Skip to content

[release/2.11] - Update triton commit and update logic to build from branch instead of triton.txt#3267

Open
jataylo wants to merge 2 commits into
ROCm:release/2.11from
jataylo:triton-internal-2.11
Open

[release/2.11] - Update triton commit and update logic to build from branch instead of triton.txt#3267
jataylo wants to merge 2 commits into
ROCm:release/2.11from
jataylo:triton-internal-2.11

Build Triton from internal branch in CI

e48db0e
Select commit
Loading
Failed to load commit list.
ROCm Repo Management API / Jenkins failed Jun 2, 2026 in 8h 58m 22s

Tests/Test Inductor/Run pytorch_inductor_1: warning in 'junit' step

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script

Error in sh step, with arguments ./test_pytorch_test.sh.

script returned exit code 1
Build log 
Build log truncated.

[2026-06-02T16:29:51.617Z] configfile: pytest.ini
[2026-06-02T16:29:51.617Z] plugins: flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, xdist-3.3.1, subtests-0.13.1, cpp-2.3.0, hypothesis-6.56.4, typeguard-4.3.0
[2026-06-02T16:29:51.617Z] collecting ... collected 289 items / 34 deselected / 255 selected
[2026-06-02T16:29:51.617Z] stepcurrent: Cannot find last run test, not skipping
[2026-06-02T16:29:51.618Z] Running 255 items in this shard: test/test_cuda.py::TestCuda::test_arithmetic_large_tensor, test/test_cuda.py::TestCuda::test_batch_norm_gather_stats, test/test_cuda.py::TestCuda::test_bincount_ext, test/test_cuda.py::TestCuda::test_caching_allocator_record_stream_oom, test/test_cuda.py::TestCuda::test_caching_pinned_memory, test/test_cuda.py::TestCuda::test_check_error, test/test_cuda.py::TestCuda::test_copy_non_blocking, test/test_cuda.py::TestCuda::test_copy_non_blocking_type_conversion, test/test_cuda.py::TestCuda::test_cublas_allow_bf16_reduced_precision_reduction_get_set, test/test_cuda.py::TestCuda::test_cublas_allow_fp16_accumulation_get_set, test/test_cuda.py::TestCuda::test_cublas_allow_fp16_reduced_precision_reduction_get_set, test/test_cuda.py::TestCuda::test_cublas_allow_tf32_get_set, test/test_cuda.py::TestCuda::test_cublas_multiple_threads_same_device, test/test_cuda.py::TestCuda::test_cublas_workspace_explicit_allocation, test/test_cuda.py::TestCuda::test_cuda_get_device_capability, test/test_cuda.py::TestCuda::test_cuda_get_device_name, test/test_cuda.py::TestCuda::test_cuda_get_device_properties, test/test_cuda.py::TestCuda::test_cuda_graph_allocator_propagates_stream, test/test_cuda.py::TestCuda::test_cuda_graph_error_options, test/test_cuda.py::TestCuda::test_cuda_graph_inference_mode, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph_exec_keep_graph_False, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph_exec_keep_graph_True, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph_keep_graph_false, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph_reset_and_recapture, test/test_cuda.py::TestCuda::test_cuda_graph_tensor_item_not_allowed, test/test_cuda.py::TestCuda::test_cuda_memory_leak_detection_propagates_errors, test/test_cuda.py::TestCuda::test_cuda_stream_protocol, test/test_cuda.py::TestCuda::test_cudart_register, test/test_cuda.py::TestCuda::test_cudnn_allow_tf32_get_set, test/test_cuda.py::TestCuda::test_cudnn_multiple_threads_same_device, test/test_cuda.py::TestCuda::test_cusparse_multiple_threads_same_device, test/test_cuda.py::TestCuda::test_device_context_manager, test/test_cuda.py::TestCuda::test_device_count_not_cached_pre_init, test/test_cuda.py::TestCuda::test_events, test/test_cuda.py::TestCuda::test_events_elapsedtime, test/test_cuda.py::TestCuda::test_fixed_cuda_assert_async, test/test_cuda.py::TestCuda::test_float32_matmul_precision_get_set, test/test_cuda.py::TestCuda::test_fp32_precision_with_tf32, test/test_cuda.py::TestCuda::test_gather_bool, test/test_cuda.py::TestCuda::test_gds_fails_in_ci, test/test_cuda.py::TestCuda::test_generic_stream_event, test/test_cuda.py::TestCuda::test_get_device_index, test/test_cuda.py::TestCuda::test_graph_capture_oom, test/test_cuda.py::TestCuda::test_graph_capture_reset_recapture, test/test_cuda.py::TestCuda::test_graph_capture_simple, test/test_cuda.py::TestCuda::test_graph_concurrent_replay, test/test_cuda.py::TestCuda::test_graph_debugdump, test/test_cuda.py::TestCuda::test_graph_error, test/test_cuda.py::TestCuda::test_graph_is_current_stream_capturing, test/test_cuda.py::TestCuda::test_graph_make_graphed_callables_same_pool, test/test_cuda.py::TestCuda::test_graph_memory_stats_and_use_result_after_destroy_graph, test/test_cuda.py::TestCuda::test_graph_optims_with_explicitly_capturable_param_groups, test/test_cuda.py::TestCuda::test_graph_record_stream, test/test_cuda.py::TestCuda::test_graph_rng_distributions, test/test_cuda.py::TestCuda::test_graph_rng_functional, test/test_cuda.py::TestCuda::test_graph_three_successive, test/test_cuda.py::TestCuda::test_graph_timing, test/test_cuda.py::TestCuda::test_graph_two_successive, test/test_cuda.py::TestCuda::test_graph_warn_if_has_zero_nodes, test/test_cuda.py::TestCuda::test_graphsafe_set_get_rng_state, test/test_cuda.py::TestCuda::test_hip_device_count, test/test_cuda.py::TestCuda::test_index_out_of_bounds_exception_cuda, test/test_cuda.py::TestCuda::test_is_pinned_no_context, test/test_cuda.py::TestCuda::test_lazy_init, test/test_cuda.py::TestCuda::test_manual_seed, test/test_cuda.py::TestCuda::test_matmul_device_mismatch, test/test_cuda.py::TestCuda::test_matmul_memory_use, test/test_cuda.py::TestCuda::test_mean_fp16, test/test_cuda.py::TestCuda::test_memory_allocation, test/test_cuda.py::TestCuda::test_memory_stats, test/test_cuda.py::TestCuda::test_memory_stats_of_multiple_generators_and_graphs, test/test_cuda.py::TestCuda::test_min_max_inits, test/test_cuda.py::TestCuda::test_multi_device_context_manager, test/test_cuda.py::TestCuda::test_multi_device_stream_context_manager, test/test_cuda.py::TestCuda::test_multinomial_ext, test/test_cuda.py::TestCuda::test_multinomial_invalid_probs_cuda, test/test_cuda.py::TestCuda::test_noncontiguous_pinned_memory, test/test_cuda.py::TestCuda::test_norm_type_conversion, test/test_cuda.py::TestCuda::test_nvtx, test/test_cuda.py::TestCuda::test_out_of_memory, test/test_cuda.py::TestCuda::test_pinned_memory_empty_cache, test/test_cuda.py::TestCuda::test_pinned_memory_use_background_threads, test/test_cuda.py::TestCuda::test_pinned_memory_with_cudaregister, test/test_cuda.py::TestCuda::test_pinned_memory_with_cudaregister_multithread, test/test_cuda.py::TestCuda::test_preferred_blas_library_settings, test/test_cuda.py::TestCuda::test_prod_large, test/test_cuda.py::TestCuda::test_randint_randomness_for_large_range, test/test_cuda.py::TestCuda::test_random_no_reused_random_states_float32, test/test_cuda.py::TestCuda::test_random_no_reused_random_states_float64, test/test_cuda.py::TestCuda::test_record_stream, test/test_cuda.py::TestCuda::test_record_stream_on_shifted_view, test/test_cuda.py::TestCuda::test_reduction_gpu_memory_accessing, test/test_cuda.py::TestCuda::test_rocm_backward_pass_guard, test/test_cuda.py::TestCuda::test_serialization_array_with_empty, test/test_cuda.py::TestCuda::test_serialization_array_with_storage, test/test_cuda.py::TestCuda::test_specify_improper_device_name, test/test_cuda.py::TestCuda::test_stream_context_manager, test/test_cuda.py::TestCuda::test_stream_event_compatibility, test/test_cuda.py::TestCuda::test_stream_event_repr, test/test_cuda.py::TestCuda::test_streaming_backwards_callback, test/test_cuda.py::TestCuda::test_streaming_backwards_multiple_streams, test/test_cuda.py::TestCuda::test_streaming_backwards_sync, test/test_cuda.py::TestCuda::test_streaming_backwards_sync_graph_root, test/test_cuda.py::TestCuda::test_streams, test/test_cuda.py::TestCuda::test_sum_fp16, test/test_cuda.py::TestCuda::test_tiny_half_norm_, test/test_cuda.py::TestCuda::test_to_cpu_blocking_by_default, test/test_cuda.py::TestCuda::test_to_numpy, test/test_cuda.py::TestCuda::test_torch_manual_seed_seeds_cuda_devices, test/test_cuda.py::TestCuda::test_type_conversions, test/test_cuda.py::TestCuda::test_uuid, test/test_cuda.py::TestCudaMallocAsync::test_allocator_backend, test/test_cuda.py::TestCudaMallocAsync::test_allocator_fuzz, test/test_cuda.py::TestCudaMallocAsync::test_allocator_memory_fraction_setting, test/test_cuda.py::TestCudaMallocAsync::test_allocator_settings, test/test_cuda.py::TestCudaMallocAsync::test_cachingAllocator_raw_alloc, test/test_cuda.py::TestCudaMallocAsync::test_clock_speed, test/test_cuda.py::TestCudaMallocAsync::test_cpp_memory_snapshot_pickle, test/test_cuda.py::TestCudaMallocAsync::test_cycles, test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used, test/test_cuda.py::TestCudaMallocAsync::test_direct_traceback, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots_free_segment_stack, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots_free_stack, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots_history_context, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots_metadata, test/test_cuda.py::TestCudaMallocAsync::test_memory_profiler_viz, test/test_cuda.py::TestCudaMallocAsync::test_memory_snapshot, test/test_cuda.py::TestCudaMallocAsync::test_memory_snapshot_forward_trace, test/test_cuda.py::TestCudaMallocAsync::test_memory_snapshot_script, test/test_cuda.py::TestCudaMallocAsync::test_memory_snapshot_skip_actions, test/test_cuda.py::TestCudaMallocAsync::test_memory_snapshot_with_cpp, test/test_cuda.py::TestCudaMallocAsync::test_notifies_oom, test/test_cuda.py::TestCudaMallocAsync::test_nvml_get_handler, test/test_cuda.py::TestCudaMallocAsync::test_power_draw, test/test_cuda.py::TestCudaMallocAsync::test_raises_oom_max_split_size_mb_setting_False, test/test_cuda.py::TestCudaMallocAsync::test_raises_oom_max_split_size_mb_setting_True, test/test_cuda.py::TestCudaMallocAsync::test_raw_amdsmi_device_count, test/test_cuda.py::TestCudaMallocAsync::test_raw_amdsmi_device_uuids, test/test_cuda.py::TestCudaMallocAsync::test_temperature, test/test_cuda.py::TestCudaMallocAsync::test_uuid_visible_devices, test/test_cuda.py::TestBlockStateAbsorption::test_additional_free_following_checkpoint, test/test_cuda.py::TestBlockStateAbsorption::test_allocate_in_thread_to_pool, test/test_cuda.py::TestBlockStateAbsorption::test_allocated_in_middle_of_segment, test/test_cuda.py::TestBlockStateAbsorption::test_assigning_back_deleter_fns_to_tensor, test/test_cuda.py::TestBlockStateAbsorption::test_check_pool_live_allocations, test/test_cuda.py::TestBlockStateAbsorption::test_expandable_segment_checkpoint_growth, test/test_cuda.py::TestBlockStateAbsorption::test_middle_allocations_contiguous, test/test_cuda.py::TestBlockStateAbsorption::test_multiple_middle_allocations, test/test_cuda.py::TestBlockStateAbsorption::test_no_triton_on_import, test/test_cuda.py::TestBlockStateAbsorption::test_resnet, test/test_cuda.py::TestBlockStateAbsorption::test_simple, test/test_cuda.py::TestBlockStateAbsorption::test_tensor_dies_after_checkpoint, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_no_use2_use_cuda_host_register_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_no_use2_use_cuda_host_register_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_no_use_use_cuda_host_register_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_no_use_use_cuda_host_register_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_use_use_cuda_host_register_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_use_use_cuda_host_register_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_False_use_memory_False_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_False_use_memory_False_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_False_use_memory_True_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_False_use_memory_True_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_True_use_memory_False_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_True_use_memory_False_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_True_use_memory_True_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_True_use_memory_True_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_False_use_memory_False_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_False_use_memory_False_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_False_use_memory_True_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_False_use_memory_True_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_True_use_memory_False_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_True_use_memory_False_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_True_use_memory_True_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_True_use_memory_True_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_unpinned_memory_use, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_unpinned_memory_use_device_to_host, test/test_cuda.py::TestMemPool::test_graph_capture_pre_capture_stream_use, test/test_cuda.py::TestMemPool::test_graph_capture_reclaim_2_streams, test/test_cuda.py::TestMemPool::test_graph_capture_reclaim_4_streams, test/test_cuda.py::TestMemPool::test_graph_capture_reclaim_shared_pool, test/test_cuda.py::TestMemPool::test_mempool_empty_cache, test/test_cuda.py::TestMemPool::test_mempool_emptycache_multithread, test/test_cuda.py::TestMemPool::test_mempool_expandable, test/test_cuda.py::TestMemPool::test_mempool_id, test/test_cuda.py::TestMemPool::test_mempool_multithread, test/test_cuda.py::TestMemPool::test_snapshot_include_traces, test/test_cuda.py::TestMemPool::test_snapshot_include_traces_correctness, test/test_cuda.py::TestGreenContext::test_greencontext_restores_stream, test/test_cuda.py::TestGDS::test_gds_read_write_tensors, test/test_cuda.py::TestCudaAutocast::test_autocast_banned, test/test_cuda.py::TestCudaAutocast::test_autocast_cat_jit, test/test_cuda.py::TestCudaAutocast::test_autocast_checkpointing, test/test_cuda.py::TestCudaAutocast::test_autocast_custom_cast_inputs, test/test_cuda.py::TestCudaAutocast::test_autocast_custom_deprecated_warning, test/test_cuda.py::TestCudaAutocast::test_autocast_custom_enabled, test/test_cuda.py::TestCudaAutocast::test_autocast_ignored_types, test/test_cuda.py::TestCudaAutocast::test_autocast_linalg_fp16, test/test_cuda.py::TestCudaAutocast::test_autocast_methods_expect_builtin_promote, test/test_cuda.py::TestCudaAutocast::test_autocast_methods_fp16, test/test_cuda.py::TestCudaAutocast::test_autocast_methods_fp32, test/test_cuda.py::TestCudaAutocast::test_autocast_nn_bf16, test/test_cuda.py::TestCudaAutocast::test_autocast_nn_fp16, test/test_cuda.py::TestCudaAutocast::test_autocast_nn_fp32, test/test_cuda.py::TestCudaAutocast::test_autocast_rnn, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_bf16, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_expect_builtin_promote, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_fp16, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_fp32, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_need_autocast_promote, test/test_cuda.py::TestCudaAutocast::test_cuda_autocast_deprecated_warning, test/test_cuda.py::TestCudaAutocast::test_rnn_packed_sequence_batch_sizes_must_be_cpu, test/test_cuda.py::TestCompileKernel::test_compile_kernel, test/test_cuda.py::TestCompileKernel::test_compile_kernel_advanced, test/test_cuda.py::TestCompileKernel::test_compile_kernel_as_custom_op, test/test_cuda.py::TestCompileKernel::test_compile_kernel_cuda_headers, test/test_cuda.py::TestCompileKernel::test_compile_kernel_custom_op_validation, test/test_cuda.py::TestCompileKernel::test_compile_kernel_dlpack, test/test_cuda.py::TestCompileKernel::test_compile_kernel_double_precision, test/test_cuda.py::TestCompileKernel::test_compile_kernel_large_shared_memory, test/test_cuda.py::TestCompileKernel::test_compile_kernel_template, test/test_cuda.py::TestFXMemoryProfiler::test_fx_memory_profiler_augmentation, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_False_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_False_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_False_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_True_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_True_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_True_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_Adagrad_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_False_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_False_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_False_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_True_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_True_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_True_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_True_fused_False_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_True_fused_False_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_True_fused_False_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_ASGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adadelta_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adamax_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_NAdam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_RAdam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_RMSprop_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Rprop_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_scaling_fused_optimizers_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_scaling_fused_optimizers_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_scaling_fused_optimizers_SGD_cuda_float32, test/test_cuda.py::TestCudaDeviceParametrizedCUDA::test_graph_external_wait_and_record_cuda
[2026-06-02T16:29:51.619Z] 
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_arithmetic_large_tensor SKIPPED [0.0005s] (was disabled due to not enough memory, but actually it always fail) [  0%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_batch_norm_gather_stats PASSED [0.3424s]    [  0%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_bincount_ext PASSED [0.8954s]               [  1%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_caching_allocator_record_stream_oom PASSED [0.0550s] [  1%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_caching_pinned_memory PASSED [1.0039s]      [  1%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_check_error PASSED [0.0016s]                [  2%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_copy_non_blocking PASSED [0.2509s]          [  2%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_copy_non_blocking_type_conversion PASSED [0.1084s] [  3%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cublas_allow_bf16_reduced_precision_reduction_get_set PASSED [0.0022s] [  3%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cublas_allow_fp16_accumulation_get_set PASSED [0.0011s] [  3%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cublas_allow_fp16_reduced_precision_reduction_get_set PASSED [0.0011s] [  4%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cublas_allow_tf32_get_set PASSED [0.0011s]  [  4%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cublas_multiple_threads_same_device PASSED [0.5303s] [  5%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cublas_workspace_explicit_allocation [W602 16:23:25.956378404 Context.cpp:541] Warning: torch.backends.cuda.preferred_blas_library is an experimental feature. If you see any error or unexpected behavior when this flag is set please file an issue on GitHub. (function operator())
[2026-06-02T16:29:51.619Z] PASSED [0.0860s] [  5%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_get_device_capability PASSED [0.0014s] [  5%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_get_device_name PASSED [0.0012s]       [  6%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_get_device_properties PASSED [0.0012s] [  6%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_graph_allocator_propagates_stream PASSED [0.0130s] [  7%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_graph_error_options PASSED [0.0110s]   [  7%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_graph_inference_mode PASSED [0.0054s]  [  7%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph SKIPPED [0.0002s] (CUDA >= 11.0 or ROCM >= 5.3 required for graphs, cuda-bindings must be installed) [  8%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph_exec_keep_graph_False SKIPPED [0.0002s] (CUDA >= 11.0 or ROCM >= 5.3 required for graphs, cuda-bindings must be installed) [  8%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph_exec_keep_graph_True SKIPPED [0.0002s] (CUDA >= 11.0 or ROCM >= 5.3 required for graphs, cuda-bindings must be installed) [  9%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph_keep_graph_false PASSED [0.0023s] [  9%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph_reset_and_recapture PASSED [0.0034s] [  9%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_graph_tensor_item_not_allowed /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/graphs.py:126: UserWarning: The CUDA Graph is empty. This usually means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at /var/lib/jenkins/pytorch/aten/src/ATen/hip/HIPGraph.cpp:148.)
[2026-06-02T16:29:51.619Z]   super().capture_end()
[2026-06-02T16:29:51.619Z] Traceback (most recent call last):
[2026-06-02T16:29:51.619Z]   File "<string>", line 17, in <module>
[2026-06-02T16:29:51.619Z]   File "<string>", line 7, in my_func
[2026-06-02T16:29:51.619Z] torch.AcceleratorError: CUDA error: operation not permitted when stream is capturing
[2026-06-02T16:29:51.619Z] Search for `hipErrorStreamCaptureUnsupported' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
[2026-06-02T16:29:51.619Z] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[2026-06-02T16:29:51.619Z] For debugging consider passing AMD_SERIALIZE_KERNEL=3
[2026-06-02T16:29:51.619Z] Device-side assertion tracking was not enabled by user.
[2026-06-02T16:29:51.619Z] PASSED [2.2566s] [ 10%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_memory_leak_detection_propagates_errors PASSED [0.0026s] [ 10%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cuda_stream_protocol PASSED [0.0025s]       [ 10%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cudart_register PASSED [0.0014s]            [ 11%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cudnn_allow_tf32_get_set PASSED [0.0011s]   [ 11%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cudnn_multiple_threads_same_device PASSED [231.4950s] [ 12%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_cusparse_multiple_threads_same_device PASSED [16.0768s] [ 12%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_device_context_manager PASSED [0.0013s]     [ 12%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_device_count_not_cached_pre_init SKIPPED [0.0002s] (requires multiple devices) [ 13%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_events PASSED [0.0513s]                     [ 13%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_events_elapsedtime PASSED [0.0018s]         [ 14%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_fixed_cuda_assert_async GPU coredump: System pipe patterns not supported in containers.
[2026-06-02T16:29:51.619Z] Falling back to file-based dump. Use custom pattern (HSA_COREDUMP_FILE) to override.
[2026-06-02T16:29:51.619Z] :0:rocdevice.cpp            :3721: 116101821466 us:  Callback: Queue 0x7ff1fd000000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
[2026-06-02T16:29:51.619Z] GPU coredump: System pipe patterns not supported in containers.
[2026-06-02T16:29:51.619Z] Falling back to file-based dump. Use custom pattern (HSA_COREDUMP_FILE) to override.
[2026-06-02T16:29:51.619Z] :0:rocdevice.cpp            :3721: 116109253782 us:  Callback: Queue 0x7f9cf9400000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
[2026-06-02T16:29:51.619Z] GPU coredump: System pipe patterns not supported in containers.
[2026-06-02T16:29:51.619Z] Falling back to file-based dump. Use custom pattern (HSA_COREDUMP_FILE) to override.
[2026-06-02T16:29:51.619Z] :0:rocdevice.cpp            :3721: 116115323694 us:  Callback: Queue 0x7fc8daa00000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
[2026-06-02T16:29:51.619Z] GPU coredump: System pipe patterns not supported in containers.
[2026-06-02T16:29:51.619Z] Falling back to file-based dump. Use custom pattern (HSA_COREDUMP_FILE) to override.
[2026-06-02T16:29:51.619Z] :0:rocdevice.cpp            :3721: 116123856676 us:  Callback: Queue 0x7f3136600000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
[2026-06-02T16:29:51.619Z] PASSED [28.2358s]   [ 14%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_float32_matmul_precision_get_set PASSED [0.0155s] [ 14%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_fp32_precision_with_tf32 PASSED [0.0014s]   [ 15%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_gather_bool PASSED [0.0566s]                [ 15%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_gds_fails_in_ci PASSED [0.0013s]            [ 16%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_generic_stream_event PASSED [0.0019s]       [ 16%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_get_device_index PASSED [0.0011s]           [ 16%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_capture_oom PASSED [0.0036s]          [ 17%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_capture_reset_recapture PASSED [0.0334s] [ 17%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_capture_simple PASSED [0.0018s]       [ 18%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_concurrent_replay SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/104055 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 18%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_debugdump PASSED [0.0078s]            [ 18%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_error PASSED [3.0027s]                [ 19%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_is_current_stream_capturing PASSED [0.0015s] [ 19%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_make_graphed_callables_same_pool PASSED [0.9835s] [ 20%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_memory_stats_and_use_result_after_destroy_graph PASSED [2.1144s] [ 20%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_optims_with_explicitly_capturable_param_groups PASSED [0.6874s] [ 20%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_record_stream PASSED [0.0342s]        [ 21%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_rng_distributions PASSED [0.3436s]    [ 21%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_rng_functional PASSED [0.0451s]       [ 21%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_three_successive PASSED [0.0171s]     [ 22%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_timing SKIPPED [0.0003s] (CUDA >= 11.0 required for external events in cuda graphs. rocm does not support external events) [ 22%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_two_successive PASSED [0.0198s]       [ 23%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graph_warn_if_has_zero_nodes PASSED [0.0013s] [ 23%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_graphsafe_set_get_rng_state PASSED [0.0048s] [ 23%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_hip_device_count PASSED [12.4575s]          [ 24%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_index_out_of_bounds_exception_cuda SKIPPED [0.0012s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [ 24%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_is_pinned_no_context PASSED [3.3983s]       [ 25%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_lazy_init PASSED [4.3908s]                  [ 25%]
[2026-06-02T16:29:51.619Z] test_cuda.py::TestCuda::test_manual_seed PASSED [0.0025s]                [ 25%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_matmul_device_mismatch PASSED [0.0023s]     [ 26%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_matmul_memory_use PASSED [0.0051s]          [ 26%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_mean_fp16 PASSED [0.0134s]                  [ 27%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_memory_allocation PASSED [0.1020s]          [ 27%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_memory_stats PASSED [0.1525s]               [ 27%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_memory_stats_of_multiple_generators_and_graphs PASSED [0.6061s] [ 28%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_min_max_inits PASSED [0.0021s]              [ 28%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_multi_device_context_manager SKIPPED [0.0002s] (only one GPU detected) [ 29%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_multi_device_stream_context_manager SKIPPED [0.0002s] (only one GPU detected) [ 29%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_multinomial_ext PASSED [0.1970s]            [ 29%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_multinomial_invalid_probs_cuda SKIPPED [0.0013s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [ 30%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_noncontiguous_pinned_memory PASSED [0.0015s] [ 30%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_norm_type_conversion PASSED [0.0286s]       [ 30%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_nvtx PASSED [0.0010s]                       [ 31%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_out_of_memory PASSED [0.0029s]              [ 31%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_pinned_memory_empty_cache PASSED [0.1900s]  [ 32%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_pinned_memory_use_background_threads PASSED [3.0241s] [ 32%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_pinned_memory_with_cudaregister PASSED [0.2407s] [ 32%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_pinned_memory_with_cudaregister_multithread PASSED [0.3348s] [ 33%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_preferred_blas_library_settings PASSED [13.6304s] [ 33%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_prod_large PASSED [0.0088s]                 [ 34%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_randint_randomness_for_large_range PASSED [1.7669s] [ 34%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_random_no_reused_random_states_float32 PASSED [1.3960s] [ 34%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_random_no_reused_random_states_float64 PASSED [1.3271s] [ 35%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_record_stream PASSED [0.0522s]              [ 35%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_record_stream_on_shifted_view SKIPPED [0.0006s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/120318 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 36%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_reduction_gpu_memory_accessing PASSED [0.0017s] [ 36%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_rocm_backward_pass_guard PASSED [0.0019s]   [ 36%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_serialization_array_with_empty PASSED [0.0046s] [ 37%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_serialization_array_with_storage PASSED [0.0035s] [ 37%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_specify_improper_device_name PASSED [0.0018s] [ 38%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_stream_context_manager PASSED [0.0012s]     [ 38%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_stream_event_compatibility PASSED [0.0021s] [ 38%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_stream_event_repr PASSED [0.0010s]          [ 39%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_streaming_backwards_callback PASSED [0.0062s] [ 39%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_streaming_backwards_multiple_streams PASSED [0.0897s] [ 40%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_streaming_backwards_sync PASSED [0.0137s]   [ 40%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_streaming_backwards_sync_graph_root PASSED [0.2572s] [ 40%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_streams PASSED [0.0013s]                    [ 41%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_sum_fp16 PASSED [0.3091s]                   [ 41%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_tiny_half_norm_ PASSED [0.0018s]            [ 41%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_to_cpu_blocking_by_default PASSED [1.1023s] [ 42%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_to_numpy PASSED [0.0137s]                   [ 42%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_torch_manual_seed_seeds_cuda_devices PASSED [0.0026s] [ 43%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_type_conversions PASSED [0.0080s]           [ 43%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCuda::test_uuid PASSED [0.0010s]                       [ 43%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_allocator_backend PASSED [4.0959s] [ 44%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_allocator_fuzz PASSED [1.1295s]  [ 44%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_allocator_memory_fraction_setting PASSED [18.6021s] [ 45%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_allocator_settings PASSED [0.0206s] [ 45%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_cachingAllocator_raw_alloc PASSED [0.0019s] [ 45%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_clock_speed PASSED [0.0017s]     [ 46%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_cpp_memory_snapshot_pickle PASSED [15.6698s] [ 46%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_cycles W0602 16:29:35.927000 14481 site-packages/torch/utils/viz/_cycles.py:59] CUDA Memory changed during GC, 512 bytes freed.
[2026-06-02T16:29:51.620Z] PASSED [0.3462s]          [ 47%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1334s] [ 47%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1255s] [ 47%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used FAILED [0.0926s] [ 47%]
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] ==================================== RERUNS ====================================
[2026-06-02T16:29:51.620Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-06-02T16:29:51.620Z] Traceback (most recent call last):
[2026-06-02T16:29:51.620Z]   File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 5135, in test_device_memory_used
[2026-06-02T16:29:51.620Z]     self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-06-02T16:29:51.620Z]   File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-06-02T16:29:51.620Z]     raise self.failureException(msg)
[2026-06-02T16:29:51.620Z] AssertionError: False is not true
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.620Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.620Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-06-02T16:29:51.620Z] Traceback (most recent call last):
[2026-06-02T16:29:51.620Z]   File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 5135, in test_device_memory_used
[2026-06-02T16:29:51.620Z]     self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-06-02T16:29:51.620Z]   File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-06-02T16:29:51.620Z]     raise self.failureException(msg)
[2026-06-02T16:29:51.620Z] AssertionError: False is not true
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.620Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.620Z] =================================== FAILURES ===================================
[2026-06-02T16:29:51.620Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-06-02T16:29:51.620Z] Traceback (most recent call last):
[2026-06-02T16:29:51.620Z]   File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 5135, in test_device_memory_used
[2026-06-02T16:29:51.620Z]     self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-06-02T16:29:51.620Z]   File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-06-02T16:29:51.620Z]     raise self.failureException(msg)
[2026-06-02T16:29:51.620Z] AssertionError: False is not true
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.620Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.620Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_cuda/test_cuda-b000dda2616d49e8.xml -
[2026-06-02T16:29:51.620Z] =========================== short test summary info ============================
[2026-06-02T16:29:51.620Z] FAILED [0.0926s] test_cuda.py::TestCudaMallocAsync::test_device_memory_used - AssertionError: False is not true
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.620Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.620Z] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
[2026-06-02T16:29:51.620Z] = 1 failed, 108 passed, 12 skipped, 34 deselected, 2 rerun in 374.46s (0:06:14) =
[2026-06-02T16:29:51.620Z] Got exit code 1
[2026-06-02T16:29:51.620Z] Retrying single test...
[2026-06-02T16:29:51.620Z] Test results will be stored in test-reports/python-pytest/test_cuda/test_cuda-993b746375f4bd85.xml
[2026-06-02T16:29:51.620Z] ============================= test session starts ==============================
[2026-06-02T16:29:51.620Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T16:29:51.620Z] cachedir: .pytest_cache
[2026-06-02T16:29:51.620Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T16:29:51.620Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T16:29:51.620Z] configfile: pytest.ini
[2026-06-02T16:29:51.620Z] plugins: flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, xdist-3.3.1, subtests-0.13.1, cpp-2.3.0, hypothesis-6.56.4, typeguard-4.3.0
[2026-06-02T16:29:51.620Z] collecting ... collected 289 items / 288 deselected / 1 selected
[2026-06-02T16:29:51.620Z] stepcurrent: skipping 142 already run items. Running only test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used
[2026-06-02T16:29:51.620Z] Running 1 items in this shard
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1533s] [100%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1139s] [100%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used FAILED [0.1042s] [100%]
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] ==================================== RERUNS ====================================
[2026-06-02T16:29:51.620Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-06-02T16:29:51.620Z] Traceback (most recent call last):
[2026-06-02T16:29:51.620Z]   File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 5135, in test_device_memory_used
[2026-06-02T16:29:51.620Z]     self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-06-02T16:29:51.620Z]   File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-06-02T16:29:51.620Z]     raise self.failureException(msg)
[2026-06-02T16:29:51.620Z] AssertionError: False is not true
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.620Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.620Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-06-02T16:29:51.620Z] Traceback (most recent call last):
[2026-06-02T16:29:51.620Z]   File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 5135, in test_device_memory_used
[2026-06-02T16:29:51.620Z]     self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-06-02T16:29:51.620Z]   File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-06-02T16:29:51.620Z]     raise self.failureException(msg)
[2026-06-02T16:29:51.620Z] AssertionError: False is not true
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.620Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.620Z] =================================== FAILURES ===================================
[2026-06-02T16:29:51.620Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-06-02T16:29:51.620Z] Traceback (most recent call last):
[2026-06-02T16:29:51.620Z]   File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 5135, in test_device_memory_used
[2026-06-02T16:29:51.620Z]     self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-06-02T16:29:51.620Z]   File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-06-02T16:29:51.620Z]     raise self.failureException(msg)
[2026-06-02T16:29:51.620Z] AssertionError: False is not true
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.620Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.620Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_cuda/test_cuda-993b746375f4bd85.xml -
[2026-06-02T16:29:51.620Z] =========================== short test summary info ============================
[2026-06-02T16:29:51.620Z] FAILED [0.1042s] test_cuda.py::TestCudaMallocAsync::test_device_memory_used - AssertionError: False is not true
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.620Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.620Z] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
[2026-06-02T16:29:51.620Z] ================== 1 failed, 288 deselected, 2 rerun in 0.40s ==================
[2026-06-02T16:29:51.620Z] Got exit code 1
[2026-06-02T16:29:51.620Z] Retrying single test...
[2026-06-02T16:29:51.620Z] Test results will be stored in test-reports/python-pytest/test_cuda/test_cuda-4adeed9dfa2d48d3.xml
[2026-06-02T16:29:51.620Z] ============================= test session starts ==============================
[2026-06-02T16:29:51.620Z] platform linux -- Python 3.12.13, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-06-02T16:29:51.620Z] cachedir: .pytest_cache
[2026-06-02T16:29:51.620Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-06-02T16:29:51.620Z] rootdir: /var/lib/jenkins/pytorch
[2026-06-02T16:29:51.620Z] configfile: pytest.ini
[2026-06-02T16:29:51.620Z] plugins: flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, xdist-3.3.1, subtests-0.13.1, cpp-2.3.0, hypothesis-6.56.4, typeguard-4.3.0
[2026-06-02T16:29:51.620Z] collecting ... collected 289 items / 288 deselected / 1 selected
[2026-06-02T16:29:51.620Z] stepcurrent: skipping 142 already run items. Running only test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used
[2026-06-02T16:29:51.620Z] Running 1 items in this shard
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1586s] [100%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1060s] [100%]
[2026-06-02T16:29:51.620Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used FAILED [0.1000s] [100%]
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] ==================================== RERUNS ====================================
[2026-06-02T16:29:51.620Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-06-02T16:29:51.620Z] Traceback (most recent call last):
[2026-06-02T16:29:51.620Z]   File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 5135, in test_device_memory_used
[2026-06-02T16:29:51.620Z]     self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-06-02T16:29:51.620Z]   File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-06-02T16:29:51.620Z]     raise self.failureException(msg)
[2026-06-02T16:29:51.620Z] AssertionError: False is not true
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.620Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.620Z] 
[2026-06-02T16:29:51.620Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.620Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-06-02T16:29:51.620Z] Traceback (most recent call last):
[2026-06-02T16:29:51.621Z]   File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 5135, in test_device_memory_used
[2026-06-02T16:29:51.621Z]     self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-06-02T16:29:51.621Z]   File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-06-02T16:29:51.621Z]     raise self.failureException(msg)
[2026-06-02T16:29:51.621Z] AssertionError: False is not true
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.621Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.621Z] =================================== FAILURES ===================================
[2026-06-02T16:29:51.621Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-06-02T16:29:51.621Z] Traceback (most recent call last):
[2026-06-02T16:29:51.621Z]   File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 5135, in test_device_memory_used
[2026-06-02T16:29:51.621Z]     self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-06-02T16:29:51.621Z]   File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-06-02T16:29:51.621Z]     raise self.failureException(msg)
[2026-06-02T16:29:51.621Z] AssertionError: False is not true
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.621Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.621Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_cuda/test_cuda-4adeed9dfa2d48d3.xml -
[2026-06-02T16:29:51.621Z] =========================== short test summary info ============================
[2026-06-02T16:29:51.621Z] FAILED [0.1000s] test_cuda.py::TestCudaMallocAsync::test_device_memory_used - AssertionError: False is not true
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] To execute this test, run the following from the base repo dir:
[2026-06-02T16:29:51.621Z]     PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-06-02T16:29:51.621Z] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
[2026-06-02T16:29:51.621Z] ================== 1 failed, 288 deselected, 2 rerun in 0.40s ==================
[2026-06-02T16:29:51.621Z] Got exit code 1
[2026-06-02T16:29:51.621Z] FAILED CONSISTENTLY: test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used
[2026-06-02T16:29:51.621Z] Stopping at first consistent failure
[2026-06-02T16:29:51.621Z] The following tests failed consistently: ['test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used']
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] FINISHED PRINTING LOG FILE of test_cuda 1/1 (test/test-reports/test_cuda_1.1_6dfce01785b6eaa5_.log)
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] Finished test_cuda 1/1 ... [2026-06-02 16:29:49.059347][116229.863918495], took 6.54min
[2026-06-02T16:29:51.621Z] test_cuda 1/1 failed!
[2026-06-02T16:29:51.621Z] Running test batch 'tests to run' cost 535.74 seconds
[2026-06-02T16:29:51.621Z] Emitting td_test_failure_stats_v2
[2026-06-02T16:29:51.621Z] /var/lib/jenkins/pytorch/tools/stats/upload_metrics.py:140: UserWarning: Not emitting metrics for td_test_failure_stats_v2. Missing repo. Please set the GITHUB_REPOSITORY environment variable to pass in this value.
[2026-06-02T16:29:51.621Z]   warn(f"Not emitting metrics for {metric_name}. {e}")
[2026-06-02T16:29:51.621Z] test_cuda 1/1 failed!
[2026-06-02T16:29:51.621Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_autograd/test_autograd-b97210cad5749b1f.xml
[2026-06-02T16:29:51.621Z] Found job id: None
[2026-06-02T16:29:51.621Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-06-02T16:29:51.621Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_autograd/test_autograd-b97210cad5749b1f.xml
[2026-06-02T16:29:51.621Z] Found job id: None
[2026-06-02T16:29:51.621Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-06-02T16:29:51.621Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_autograd/test_autograd-b97210cad5749b1f.xml
[2026-06-02T16:29:51.621Z] Found job id: None
[2026-06-02T16:29:51.621Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-06-02T16:29:51.621Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_autograd/test_autograd-b97210cad5749b1f.xml
[2026-06-02T16:29:51.621Z] Found job id: None
[2026-06-02T16:29:51.621Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-06-02T16:29:51.621Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_autograd/test_autograd-b97210cad5749b1f.xml
[2026-06-02T16:29:51.621Z] Found job id: None
[2026-06-02T16:29:51.621Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-06-02T16:29:51.621Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_autograd/test_autograd-b97210cad5749b1f.xml
[2026-06-02T16:29:51.621Z] Found job id: None
[2026-06-02T16:29:51.621Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-06-02T16:29:51.621Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_autograd/test_autograd-b97210cad5749b1f.xml
[2026-06-02T16:29:51.621Z] Found job id: None
[2026-06-02T16:29:51.621Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-06-02T16:29:51.621Z] GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
[2026-06-02T16:29:51.621Z] Uploading artifacts took 0.00 seconds
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] real	9m0.850s
[2026-06-02T16:29:51.621Z] user	138m41.692s
[2026-06-02T16:29:51.621Z] sys	3m3.225s
[2026-06-02T16:29:51.621Z] + sccache_epilogue
[2026-06-02T16:29:51.621Z] ::group::Sccache Compilation Log
[2026-06-02T16:29:51.621Z] =================== sccache compilation log ===================
[2026-06-02T16:29:51.621Z] + echo '::group::Sccache Compilation Log'
[2026-06-02T16:29:51.621Z] + echo '=================== sccache compilation log ==================='
[2026-06-02T16:29:51.621Z] + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /root/sccache_error.log
[2026-06-02T16:29:51.621Z] + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
[2026-06-02T16:29:51.621Z] + sccache --show-stats
[2026-06-02T16:29:51.621Z] =========== If your build fails, please take a look at the log above for possible reasons ===========
[2026-06-02T16:29:51.621Z] Compile requests                    308
[2026-06-02T16:29:51.621Z] Compile requests executed           112
[2026-06-02T16:29:51.621Z] Cache hits                           24
[2026-06-02T16:29:51.621Z] Cache hits (Assembler)                8
[2026-06-02T16:29:51.621Z] Cache hits (C/C++)                   16
[2026-06-02T16:29:51.621Z] Cache misses                         88
[2026-06-02T16:29:51.621Z] Cache misses (Assembler)             12
[2026-06-02T16:29:51.621Z] Cache misses (C/C++)                 70
[2026-06-02T16:29:51.621Z] Cache misses (HIP)                    6
[2026-06-02T16:29:51.621Z] Cache hits rate                   21.43 %
[2026-06-02T16:29:51.621Z] Cache hits rate (Assembler)       40.00 %
[2026-06-02T16:29:51.621Z] Cache hits rate (C/C++)           18.60 %
[2026-06-02T16:29:51.621Z] Cache hits rate (HIP)              0.00 %
[2026-06-02T16:29:51.621Z] Cache timeouts                        0
[2026-06-02T16:29:51.621Z] Cache read errors                     0
[2026-06-02T16:29:51.621Z] Forced recaches                       0
[2026-06-02T16:29:51.621Z] Cache write errors                    0
[2026-06-02T16:29:51.621Z] Cache errors                          0
[2026-06-02T16:29:51.621Z] Compilations                         88
[2026-06-02T16:29:51.621Z] Compilation failures                  0
[2026-06-02T16:29:51.621Z] Non-cacheable compilations            0
[2026-06-02T16:29:51.621Z] Non-cacheable calls                   7
[2026-06-02T16:29:51.621Z] Non-compilation calls               189
[2026-06-02T16:29:51.621Z] Unsupported compiler calls            0
[2026-06-02T16:29:51.621Z] Average cache write               0.001 s
[2026-06-02T16:29:51.621Z] Average compiler                  8.404 s
[2026-06-02T16:29:51.621Z] Average cache read hit            0.000 s
[2026-06-02T16:29:51.621Z] Failed distributed compilations       0
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] Non-cacheable reasons:
[2026-06-02T16:29:51.621Z] -E                                    7
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] Cache location                  Local disk: "/root/.cache/sccache"
[2026-06-02T16:29:51.621Z] Use direct/preprocessor mode?   yes
[2026-06-02T16:29:51.621Z] Version (client)                0.13.0
[2026-06-02T16:29:51.621Z] Cache size                            6 MiB
[2026-06-02T16:29:51.621Z] Max cache size                       10 GiB
[2026-06-02T16:29:51.621Z] + sccache --stop-server
[2026-06-02T16:29:51.621Z] Stopping sccache server...
[2026-06-02T16:29:51.621Z] Compile requests                    308
[2026-06-02T16:29:51.621Z] Compile requests executed           112
[2026-06-02T16:29:51.621Z] Cache hits                           24
[2026-06-02T16:29:51.621Z] Cache hits (Assembler)                8
[2026-06-02T16:29:51.621Z] Cache hits (C/C++)                   16
[2026-06-02T16:29:51.621Z] Cache misses                         88
[2026-06-02T16:29:51.621Z] Cache misses (Assembler)             12
[2026-06-02T16:29:51.621Z] Cache misses (C/C++)                 70
[2026-06-02T16:29:51.621Z] Cache misses (HIP)                    6
[2026-06-02T16:29:51.621Z] Cache hits rate                   21.43 %
[2026-06-02T16:29:51.621Z] Cache hits rate (Assembler)       40.00 %
[2026-06-02T16:29:51.621Z] Cache hits rate (C/C++)           18.60 %
[2026-06-02T16:29:51.621Z] Cache hits rate (HIP)              0.00 %
[2026-06-02T16:29:51.621Z] Cache timeouts                        0
[2026-06-02T16:29:51.621Z] Cache read errors                     0
[2026-06-02T16:29:51.621Z] Forced recaches                       0
[2026-06-02T16:29:51.621Z] Cache write errors                    0
[2026-06-02T16:29:51.621Z] Cache errors                          0
[2026-06-02T16:29:51.621Z] Compilations                         88
[2026-06-02T16:29:51.621Z] Compilation failures                  0
[2026-06-02T16:29:51.621Z] Non-cacheable compilations            0
[2026-06-02T16:29:51.621Z] Non-cacheable calls                   7
[2026-06-02T16:29:51.621Z] Non-compilation calls               189
[2026-06-02T16:29:51.621Z] Unsupported compiler calls            0
[2026-06-02T16:29:51.621Z] Average cache write               0.001 s
[2026-06-02T16:29:51.621Z] Average compiler                  8.404 s
[2026-06-02T16:29:51.621Z] Average cache read hit            0.000 s
[2026-06-02T16:29:51.621Z] Failed distributed compilations       0
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] Non-cacheable reasons:
[2026-06-02T16:29:51.621Z] -E                                    7
[2026-06-02T16:29:51.621Z] 
[2026-06-02T16:29:51.621Z] Cache location                  Local disk: "/root/.cache/sccache"
[2026-06-02T16:29:51.621Z] Use direct/preprocessor mode?   yes
[2026-06-02T16:29:51.621Z] Version (client)                0.13.0
[2026-06-02T16:29:51.621Z] Cache size                            6 MiB
[2026-06-02T16:29:51.621Z] Max cache size                       10 GiB
[2026-06-02T16:29:51.621Z] ::endgroup::
[2026-06-02T16:29:51.621Z] + echo ::endgroup::
[2026-06-02T16:29:51.621Z] + cp -RT test/test-reports /host_workspace/pytorch_reports
[2026-06-02T16:29:51.621Z] + chmod -R 777 /host_workspace/pytorch_log /host_workspace/pytorch_reports
[2026-06-02T16:29:51.621Z] + git clean -fdx
[2026-06-02T16:29:51.621Z] Removing .additional_ci_files/
[2026-06-02T16:29:51.621Z] Removing .pytest_cache/
[2026-06-02T16:29:51.621Z] Removing build/
[2026-06-02T16:29:51.621Z] Removing dist/
[2026-06-02T16:29:51.621Z] Removing test/.pytorch-disabled-tests.json
[2026-06-02T16:29:51.621Z] Removing test/__pycache__/
[2026-06-02T16:29:51.621Z] Removing test/autograd/__pycache__/
[2026-06-02T16:29:51.621Z] Removing test/gpucore.54904.gpu
[2026-06-02T16:29:51.621Z] Removing test/gpucore.55160.gpu
[2026-06-02T16:29:51.621Z] Removing test/gpucore.55704.gpu
[2026-06-02T16:29:51.621Z] Removing test/gpucore.56051.gpu
[2026-06-02T16:29:51.621Z] Removing test/test-reports/
[2026-06-02T16:29:51.621Z] Removing test/torch_compile_debug/
[2026-06-02T16:29:51.621Z] Removing test_artifacts.zip
[2026-06-02T16:29:51.621Z] Removing tools/__pycache__/
[2026-06-02T16:29:51.621Z] Removing tools/stats/__pycache__/
[2026-06-02T16:29:51.621Z] Removing tools/testing/__pycache__/
[2026-06-02T16:29:51.621Z] Removing tools/testing/target_determination/__pycache__/
[2026-06-02T16:29:51.621Z] Removing tools/testing/target_determination/heuristics/__pycache__/
[2026-06-02T16:29:51.621Z] Removing torch-2.11.0+gite48db0e-cp312-cp312-linux_x86_64.whl

Output truncated.

Details

  • Kill older PR Builds (1.6 sec)
  • Initialize (2 hr 32 min)
    • Download CI scripts (31 sec)
    • Checkout Pytorch (1 min 12 sec)
    • Check base Docker image existence (7.5 sec)
    • Build Docker image (1 hr 17 min)
    • Build PyTorch (1 hr 11 min)
  • Tests (6 hr 26 min)
    • Test PyTorch (7 ms)
      • Test PyTorch (3 hr 7 min)
        • Run pytorch_test_1 (1 hr 49 min)
        • Run pytorch_test_2 (1 hr 17 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_test_2 failed - logs
          Unstable: 3 tests failed - logs
    • Test Distributed (9 ms)
      • Test Distributed (6 hr 25 min)
        • Run pytorch_distributed_1 (3 hr 24 min)
        • Run pytorch_distributed_2 (3 hr 0 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_distributed_2 failed - logs
          Unstable: 4 tests failed - logs
    • Test Inductor (10 ms)
      • Test Inductor (2 hr 49 min)
        • Run pytorch_inductor_1 (2 hr 49 min)
          Error: Found 1 failure(s) in pytorch_reports/python-pytest/inductor.test_aot_inductor/inductor.test_aot_inductor-5b7edd6d93a3a845.xml report - logs
          Error: Some tests are failed or errored - logs
          Error: pytorch_inductor_1 failed - logs
          Unstable: 1 tests failed - logs
    • Test PyTorch Slow (13 ms)
      • Test PyTorch Slow (11 sec)
    • Microbenchmark (21 sec)
      • Microbenchmark (11 sec)
  • Post Build (2 sec)
  • Declarative: Post Actions (4.9 sec)
View more details on ROCm Repo Management API