feat(clp-package): Add Python package config, metadata-DB column, and scheduler handoff for the compression coordinator.

[LinZhihao-723]

Description

This PR adds the Python-side support that the compression coordinator needs: the package configuration for the coordinator and its Spider connection, the spider_id column the coordinator uses to track and recover Spider jobs, and a handoff in the legacy compression scheduler so it stops competing for the jobs the coordinator now owns.

The configuration added here mirrors the spider and compression_coordinator sections introduced on the Rust side in #2417, so it depends on #2417 for the config to be meaningful. It does not strictly build on top of #2417, though: none of these changes reference or require that PR's code, and this PR can be reviewed and merged independently.

Package configuration

Adds Spider and CompressionCoordinator config models (plus their nested SpiderResourceGroup and PollingBackoff) to clp_config.py, and exposes them as optional spider and compression_coordinator fields on ClpConfig (both default None).

• The models mirror the Rust structs field-for-field, and the Python field names match the Rust serde keys exactly, so the same YAML round-trips between the two.
• CompressionCoordinator's defaults (polling interval, result-poll backoff, task retry limits, DB pool size, termination and commit-task timeouts) match the Rust Default implementation.
• SpiderResourceGroup and PollingBackoff carry no field defaults of their own; their values are supplied where they are initialized inside CompressionCoordinator.
• Adds a validate_compression_coordinator_config model validator: when compression_coordinator is configured, it requires the clp-s storage engine and a non-null spider config. It intentionally does not couple the coordinator to spider_scheduler or to Spider orchestration, so the coordinator can run alongside a Celery-based compression scheduler.

Metadata-DB schema

Adds a spider_id BIGINT UNSIGNED NULL DEFAULT NULL column and a JOB_SPIDER_ID index to the compression_jobs table. The coordinator writes the Spider job ID here once it submits a job to Spider, and on startup it recovers in-flight jobs by selecting the RUNNING rows whose spider_id is non-null. The column is nullable and defaults to NULL because a job has no Spider job ID until the coordinator submits it.

Compression scheduler handoff

When compression_coordinator is configured, the legacy Celery-based compression scheduler skips S3_OBJECT_METADATA jobs: it logs the skip and leaves the job in PENDING so the coordinator picks it up. The coordinator handles only S3_OBJECT_METADATA input, so this cleanly partitions the work — all other input types (FS, S3) continue to flow through the legacy scheduler unchanged.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.

Summary by CodeRabbit

• New Features

  • Added configuration support for Spider orchestration and the Compression Coordinator, including resource groups, polling, retry, pooling, and timeout settings.
  • Added validation to ensure compatible storage and orchestration settings are provided together.
  • Added tracking for Spider assignment and dispatch times in compression jobs.

• Bug Fixes

  • Compression jobs remain pending for coordinator-managed processing instead of being processed prematurely.
  • Job update timestamps now refresh automatically when records change.

[coderabbitai]

Walkthrough

Adds Spider and compression coordinator configuration with validation, extends the compression jobs schema for coordinator tracking, and defers S3 metadata jobs to coordinator handling when enabled.

Changes

Spider compression coordinator

Layer / File(s)	Summary
Coordinator configuration and validation components/clp-py-utils/clp_py_utils/clp_config.py	Adds nested Spider and compression coordinator models, exposes optional ClpConfig fields, and requires clp-s storage plus Spider configuration when the coordinator is enabled.
Compression job schema updates components/clp-py-utils/clp_py_utils/initialize-orchestration-db.py	Adds automatic update timestamps, Spider and dispatch fields, and a spider_id index to the compression jobs table.
Scheduler coordinator handoff components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py	Leaves S3 metadata jobs pending for coordinator handling when configured; otherwise preserves the existing processing path.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• y-scope/clp#2375: Changes the same scheduler entry point’s configuration deserialization.
• y-scope/clp#2405: Adds compression-coordinator scaffolding for running and recovering S3 compression jobs.
• y-scope/clp#2417: Adds the coordinator binary that reads compression jobs and dispatches them to Spider.

Suggested reviewers: junhaoliao

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main changes: Python config support, a database column update, and scheduler handoff for the compression coordinator.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/clp-py-utils/clp_py_utils/initialize-orchestration-db.py`:
- Around line 71-87: Extend the existing compression_jobs migration logic in
initialize-orchestration-db.py beyond the current JOB_START_TIME_STATUS
alteration: add idempotent migrations for spider_id, dispatch_time,
JOB_SPIDER_ID, and the revised update_time definition. Ensure upgraded tables
receive the same columns, index, and update behavior declared by the CREATE
TABLE schema without affecting fresh installations.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: f7503cb9-9232-4b84-8dd8-f5769271af3d

📥 Commits

Reviewing files that changed from the base of the PR and between 8f68cd9 and 6f9f455.

📒 Files selected for processing (3)

• components/clp-py-utils/clp_py_utils/clp_config.py
• components/clp-py-utils/clp_py_utils/initialize-orchestration-db.py
• components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py

[coderabbitai]

Actionable comments posted: 1

♻️ Duplicate comments (1)

components/clp-py-utils/clp_py_utils/initialize-orchestration-db.py (1)

71-87: 🗄️ Data Integrity & Integration | 🟠 Major

Migrate existing compression_jobs tables as well.

CREATE TABLE IF NOT EXISTS does not update existing deployments, and the only ALTER TABLE below adds JOB_START_TIME_STATUS. Upgraded databases will still lack spider_id, dispatch_time, JOB_SPIDER_ID, and the revised update_time behaviour, causing coordinator reads or writes to fail. Add idempotent migrations for each schema change.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/clp-py-utils/clp_py_utils/initialize-orchestration-db.py` around
lines 71 - 87, Update the compression_jobs schema initialization and migration
logic around the CREATE TABLE and existing ALTER TABLE statements to add
idempotent migrations for spider_id, dispatch_time, JOB_SPIDER_ID, and the
revised update_time definition. Ensure existing tables receive these columns,
index, and update behavior without failing when already applied, while
preserving the current creation path for new tables.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/clp-py-utils/clp_py_utils/clp_config.py`:
- Around line 861-862: Update transform_for_container() to apply the same
container hostname and port mapping used for spider_scheduler to the spider
endpoint, preserving the configured external behavior only if that is explicitly
intended and documented. Ensure containerized CompressionCoordinator connections
use the transformed Spider service endpoint instead of the default
localhost:6000.

---

Duplicate comments:
In `@components/clp-py-utils/clp_py_utils/initialize-orchestration-db.py`:
- Around line 71-87: Update the compression_jobs schema initialization and
migration logic around the CREATE TABLE and existing ALTER TABLE statements to
add idempotent migrations for spider_id, dispatch_time, JOB_SPIDER_ID, and the
revised update_time definition. Ensure existing tables receive these columns,
index, and update behavior without failing when already applied, while
preserving the current creation path for new tables.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2d67628f-c429-4e36-b976-56c12a0f0af3

📥 Commits

Reviewing files that changed from the base of the PR and between 8f68cd9 and 6f9f455.

📒 Files selected for processing (3)

• components/clp-py-utils/clp_py_utils/clp_config.py
• components/clp-py-utils/clp_py_utils/initialize-orchestration-db.py
• components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py