Minimize Dask resource acquisition in cudf_polars tests

[mroeschke]

Description

The motivation is to help alleviate potential CI issues due to Dask/Ray resource spin-up in cudf_polars tests, starting with test_dask.py

• Removes test_scan, test_filter, test_group_by, test_join, and test_empty_dataframe as they already have coverage in other test where we parameterize over engine
• For tests needing an instantiated DaskEngine, uses the session-scoped DaskEngine already created in conftest.py
• For tests exercising DaskEngine construction, uses a module-scoped dask_client with a LocalCluster (tests that use this are just testing engine properties so just uses 1 worker)

Overall, this PR reduces the 5 Dask clusters allocated to just 2

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 41941576-631f-4581-8e79-929bf6c84703

📥 Commits

Reviewing files that changed from the base of the PR and between 3cfe15b and a0c3541.

📒 Files selected for processing (2)

• python/cudf_polars/tests/conftest.py
• python/cudf_polars/tests/streaming/test_dask.py

---

📝 Walkthrough

Summary by CodeRabbit

• Tests
  • Enhanced Dask engine test configuration and fixtures to improve test isolation and reliability.
  • Reorganized test structure for better resource management and clearer test organization.

Walkthrough

This PR refactors Dask engine test fixtures to support both centralized configuration and explicit client injection. It adds a shared dask_engine fixture in conftest.py and restructures test_dask.py to use a dedicated dask_client fixture for tests that construct engines directly, while deprecating the module-scoped engine fixture.

Changes

Dask Engine Fixture Refactoring

Layer / File(s)	Summary
Shared dask_engine fixture in conftest python/cudf_polars/tests/conftest.py	Adds DaskEngine import, defines dask_engine fixture that configures a session-scoped DaskEngine with configure_streaming_engine, and extends pytest_generate_tests to restrict _engine_param parametrization to "dask" backend when dask_engine is requested.
test_dask.py fixture and test rewiring python/cudf_polars/tests/streaming/test_dask.py	Removes module-scoped engine fixture and introduces dask_client fixture for explicit client creation; rewires engine-construction tests to accept dask_client and pass it to DaskEngine.from_options(); updates the reset_engine fixture to depend on dask_client; migrates existing engine-behavior tests to use the new shared dask_engine fixture; and updates test_reset_after_shutdown_raises to construct engine with injected client.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

• rapidsai/cudf#22493: Refactors engine parametrization and fixture wiring machinery, including _engine_param and engine-selection behavior in conftest.py that directly parallels this PR's updates.

Suggested labels

Python, cudf-polars

Suggested reviewers

• pentschev

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately captures the main objective of minimizing Dask resource acquisition in cudf_polars tests, which is the primary focus of the changeset.
Description check	✅ Passed	The description is directly related to the changeset, providing clear motivation, specific changes made, and quantifiable impact (5 clusters reduced to 2).
Docstring Coverage	✅ Passed	Docstring coverage is 86.67% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}