Run conda, cudf_polars CI tests with Ray

[mroeschke]

Description

Follow up to #22381

Adds depends_on_ray to test_python_other so the conda-python-tests-other job, which runs cudf_polars tests too, exercises the RayEngine as well.

Also adds depends_on_ray to the all so ray is included in the conda/environments/ yamls

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 064b05f7-3f0d-4c6b-a8d0-64a920fb66ae

📥 Commits

Reviewing files that changed from the base of the PR and between 9c8c1ea and 4d2e0f8.

📒 Files selected for processing (6)

• .github/workflows/pr.yaml
• conda/environments/all_cuda-129_arch-aarch64.yaml
• conda/environments/all_cuda-129_arch-x86_64.yaml
• conda/environments/all_cuda-131_arch-aarch64.yaml
• conda/environments/all_cuda-131_arch-x86_64.yaml
• dependencies.yaml

✅ Files skipped from review due to trivial changes (4)

• conda/environments/all_cuda-129_arch-aarch64.yaml
• conda/environments/all_cuda-131_arch-x86_64.yaml
• conda/environments/all_cuda-131_arch-aarch64.yaml
• conda/environments/all_cuda-129_arch-x86_64.yaml

---

📝 Walkthrough

Summary by CodeRabbit

• Chores
  • Added Ray (ray-default>=2.55.1) as a dependency across all supported CUDA versions and architectures to ensure distributed computing is available in build/test environments.
  • Updated test/dependency matrices and CI job runtime settings to accommodate the change, including enhanced container runtime limits (tracing, shared memory, and file descriptors).

Walkthrough

Declared a depends_on_ray component and added ray-default>=2.55.1 to generated Conda environment files for CUDA 12.9 and 13.1 (aarch64 and x86_64). Also added container-options to the conda-python-other-tests CI job invocation; the job script is unchanged.

Changes

Ray Dependency Addition

Layer / File(s)	Summary
Dependency Configuration dependencies.yaml	Added depends_on_ray to files.all.includes and files.test_python_other.includes and defined depends_on_ray with ray-default>=2.55.1 and outputs.
Generated Conda Environments conda/environments/all_cuda-129_arch-aarch64.yaml, conda/environments/all_cuda-129_arch-x86_64.yaml, conda/environments/all_cuda-131_arch-aarch64.yaml, conda/environments/all_cuda-131_arch-x86_64.yaml	Each generated environment adds ray-default>=2.55.1 to its dependencies list.
CI Job Container Options .github/workflows/pr.yaml	conda-python-other-tests job now passes container-options (CAP_SYS_PTRACE, --shm-size=8g, --ulimit=nofile=1000000:1000000) to the shared workflow; ci/test_python_other.sh remains the job script.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'Run conda, cudf_polars CI tests with Ray' directly and clearly summarizes the main change: adding Ray support to conda and cudf_polars CI tests.
Description check	✅ Passed	The description is directly related to the changeset, explaining that it adds depends_on_ray to test_python_other and all configurations to enable Ray in conda environments and tests.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[mroeschke]

@pentschev confirming I should add these flags for unit tests with conda like you mentioned in https://github.com/rapidsai/cudf/pull/22381/changes#r3196736965?

[pentschev]

Does this run also multi-rank RapidsMPF tests? It looks like so, for those cases the answer is always yes. Single-rank tests don't use UCX communicator, and in those cases this would not be needed.

[mroeschke]

Does this run also multi-rank RapidsMPF tests?

Correct, yeah these end up running the same, bare ci/run_cudf_polars_pytests.sh as the wheels tests which IIRC runs the multi-rank tests

[pentschev]

Does this run also multi-rank RapidsMPF tests? It looks like so, for those cases the answer is always yes. Single-rank tests don't use UCX communicator, and in those cases this would not be needed.

[mroeschke]

/merge