Run the cudf-polars test suite against DaskEngine and RayEngine#22381
Run the cudf-polars test suite against DaskEngine and RayEngine#22381madsbk wants to merge 16 commits intorapidsai:mainfrom
DaskEngine and RayEngine#22381Conversation
madsbk
added
improvement
github-actions
Bot
added
Python
madsbk
force-pushed
the
engine_reset-test-dask-and-ray
branch
3 times, most recently
from
7bb501d to
4c5b5da
Compare
madsbk
force-pushed
the
engine_reset-test-dask-and-ray
branch
from
d088a13 to
3fccdf3
Compare
madsbk
force-pushed
the
engine_reset-test-dask-and-ray
branch
from
3fccdf3 to
b294bf8
Compare
| # (rapidsmpf compatibility already validated in rapidsmpf CI) | ||
| matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))])) | ||
| build_type: pull-request | ||
| container-options: "--cap-add CAP_SYS_PTRACE --shm-size=8g --ulimit=nofile=1000000:1000000" |
There was a problem hiding this comment.
These changes are necessary to provide enough resources for UCX. For reference, we already do the same for UCXX and RapidsMPF, both need it for the same reason:
| @pytest.mark.parametrize("max_rows_per_partition", [3, 9]) | ||
| def test_join_conditional(reverse, max_rows_per_partition, streaming_engine_factory): | ||
| streaming_engine = streaming_engine_factory( | ||
| def test_join_conditional(reverse, max_rows_per_partition, spmd_engine_factory): |
There was a problem hiding this comment.
Why are these spmd only? If it's just for the pytest.warns, we should probably just make that a contextlib.nullcontext() when we have a non-spmd engine.
| # The metadata being asserted on is determined by IR lowering | ||
| # (engine-agnostic), so SPMD coverage is sufficient. |
There was a problem hiding this comment.
Is using spmd_engine_factory your general recommendation for this these types of tests that we're really, really sure should be engine-agnostic?
| common: | ||
| - output_types: [conda, requirements, pyproject] | ||
| packages: | ||
| - ray>=2.0 |
There was a problem hiding this comment.
We don't need to hold this PR for this, but perhaps we might want to pin to >=2.55.1 like we do in RapidsMPF. Only 2.55.1 support aarch64 and Python 3.14. Although this condition will satisfy version 2.55.1, it's best that we know we're testing with the minimal supported version for everything we care about.
…df into engine_reset-test-dask-and-ray
5 participants
Builds on the cached
streaming_enginesfixture from #22364, which amortizes SPMD bootstrap via_reset(), and extends the same pattern to Dask and Ray.With this change, the test matrix runs against:
["in-memory", "spmd", "spmd-small", "dask", "ray"]subject to package availability and
rrungating.We might change the different setups later, but for now CI runs:
SPMDEngine"medium","small"DaskEngine"medium"RayEngine"medium"