feat(compression-coordinator): Introduce max_concurrent_tasks config to limit job-handler concurrency. - #2435
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe change adds ChangesCompression coordinator concurrency
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CompressionCoordinator
participant JobDatabase
participant JobHandler
CompressionCoordinator->>CompressionCoordinator: Check semaphore capacity
CompressionCoordinator->>JobDatabase: Fetch bounded pending jobs
CompressionCoordinator->>JobDatabase: Mark selected jobs as dispatched
CompressionCoordinator->>JobHandler: Spawn handlers with retained permits
JobHandler->>CompressionCoordinator: Release permit on completion
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bill-hbrhbr
changed the title
max_concurrent_tasks config to limit job-handler task concurrency.
Bill-hbrhbr
changed the title
max_concurrent_tasks config to limit job-handler task concurrency.max_concurrent_tasks config to limit job-handler concurrency.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/clp-py-utils/clp_py_utils/clp_config.py`:
- Line 798: Add a shared Tokio semaphore upper-bound validation for
max_concurrent_tasks in components/clp-py-utils/clp_py_utils/clp_config.py:798
and components/clp-rust-utils/src/clp_config/package/config.rs:488, preserving
the existing positive-value validation. In
components/compression-coordinator/src/coordination.rs:137-138, validate the
configured value before constructing job_handler_sem and return the established
configuration error when it exceeds the supported limit.
In `@components/compression-coordinator/src/coordination.rs`:
- Around line 49-52: Update the recovery admission logic around the
coordinator’s recovered-handler tracking so every recovered handler contributes
to the active count, including handlers that do not hold a permit. Prevent new
jobs from being admitted until the total active recovered and newly started
handlers is below max_concurrent_tasks, and revise the recovery documentation
near the constructor to describe this limit-enforced behavior.
- Around line 270-275: Update the pending-job refill flow around
fetch_new_job_rows so it retrieves eligible jobs in bounded pages rather than
loading and retaining the entire backlog in pending_job_queue. Add durable
paging state, such as a cursor that advances only after successful processing or
equivalent recovery-safe state, and preserve first-fetch recovery semantics so
retries resume without skipping jobs or causing unbounded memory growth.
In `@components/package-template/src/etc/clp-config.template.json.yaml`:
- Line 72: Update the commented max_concurrent_tasks configuration entry in the
template to explicitly state that its value must be greater than zero,
distinguishing it from compression_scheduler.max_concurrent_tasks_per_job where
zero disables the limit.
- Line 67: Update the compression coordinator configuration note near the
“Compression coordinator config” comment to document every runtime prerequisite:
`spider`, `logs_input.type: s3`, and `archive_output.storage.type: s3`; clarify
that the related template examples must use these required S3 settings rather
than `fs` when enabling the configuration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 8fd3f43d-d433-437d-ab55-235ab39cfb5f
📒 Files selected for processing (4)
components/clp-py-utils/clp_py_utils/clp_config.pycomponents/clp-rust-utils/src/clp_config/package/config.rscomponents/compression-coordinator/src/coordination.rscomponents/package-template/src/etc/clp-config.template.json.yaml
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
components/compression-coordinator/src/coordination.rs (1)
294-300: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winRecord a job as dispatched only after handler creation succeeds.
dispatched_job_ids.push(job_id)runs before configuration deserialisation andcreate_job_handle. If either operation fails,runstill passes the ID tomark_jobs_dispatched. AnError::UnsupportedInputConfigjob therefore receivesdispatch_timewithout a coordinator handler, which removes it from the subsequent pending-job query and can interfere with the documented legacy-scheduler hand-off.Move the push after
create_job_handlereturnsOk:Proposed fix
let job_id = job_row.id; - dispatched_job_ids.push(job_id); let clp_io_config: ClpIoConfig = match BrotliMsgpack::deserialize(&job_row.serialized_clp_io_config) { Ok(clp_io_config) => clp_io_config, Err(e) => { ... } }; let Ok(job_handle) = self.create_job_handle(job_id, clp_io_config).await else { continue; }; +dispatched_job_ids.push(job_id);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/compression-coordinator/src/coordination.rs` around lines 294 - 300, Move dispatched_job_ids.push(job_id) in the run flow to after configuration deserialization and successful create_job_handle completion, so only jobs with a coordinator handler are marked dispatched; preserve failure handling without passing failed job IDs to mark_jobs_dispatched.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/package-template/src/etc/clp-config.template.json.yaml`:
- Line 67: Update the compression coordinator configuration note near the
existing logs_input.type and spider requirements to also state that
archive_output.storage.type must be "s3". Keep the documentation aligned with
the coordinator prerequisites and the archive output setting shown in the
template.
---
Outside diff comments:
In `@components/compression-coordinator/src/coordination.rs`:
- Around line 294-300: Move dispatched_job_ids.push(job_id) in the run flow to
after configuration deserialization and successful create_job_handle completion,
so only jobs with a coordinator handler are marked dispatched; preserve failure
handling without passing failed job IDs to mark_jobs_dispatched.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: b74354c7-b2d8-4659-a5ac-82902ed1fabb
📒 Files selected for processing (3)
components/compression-coordinator/src/coordination.rscomponents/compression-coordinator/src/error.rscomponents/package-template/src/etc/clp-config.template.json.yaml
| job_polling_interval: Duration, | ||
| cancellation_token: CancellationToken, | ||
| job_handler_sem: Arc<Semaphore>, | ||
| pending_job_queue: VecDeque<PendingJobRowProjection>, |
There was a problem hiding this comment.
I'm not sure if we need this. Can you explain why we can't do the following instead:
- On the main loop, fetch compression jobs with a limit and ID ordering (sth like
SELECT * FROM t ORDER BY id ASC LIMIT 100;) - Get a permit before spawning the coroutine. The permit automatially drops itself if the coroutine exits, aborts, or crashed.
- Let the semaphore to block coroutine creation implicitly. The main loop may be blocked, which is fine because it shouldn't push more jobs into Spider.
This should lead to the behavior we expect iiuc.
|
@coderabbitai review |
|
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
components/compression-coordinator/src/coordination.rs (2)
46-49: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUpdate the constructor docstring to cover both recovery categories.
The docstring states only that recovery covers jobs that are still
Runningwith a Spider job ID.recover_previous_jobsalso resumesPendingjobs whosedispatch_timeis populated. The docstring at Lines 204-217 already describes both categories.📝 Proposed doc fix
- /// On construction, this begins recovering all compression jobs that a previous coordinator - /// instance had already submitted to Spider (those still [`CompressionJobStatus::Running`] with - /// a Spider job ID). During the restart phase, no concurrency limit is imposed, so all - /// recovered jobs are resumed immediately. + /// On construction, this begins recovering all compression jobs that a previous coordinator + /// instance had already claimed: jobs still [`CompressionJobStatus::Running`] with a Spider job + /// ID, and jobs still [`CompressionJobStatus::Pending`] with a populated `dispatch_time`. + /// During the restart phase, no concurrency limit is imposed, so all recovered jobs are resumed + /// immediately.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/compression-coordinator/src/coordination.rs` around lines 46 - 49, Update the constructor documentation describing recovery to include both categories handled by recover_previous_jobs: Running jobs with a Spider job ID and Pending jobs with a populated dispatch_time. Preserve the existing note that recovered jobs resume immediately without a concurrency limit.
323-342: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winRecord a job as dispatched only after the handler is spawned.
Line 330 pushes
job_idintodispatched_job_idsbefore deserialization and handle creation. Both later steps cancontinue.Two consequences follow:
create_job_handlereturnsError::UnsupportedInputConfigfor inputs that the legacy Celery scheduler must handle. The job staysPending, butmark_jobs_dispatchednow stampsdispatch_timeon it. On every later restart,fetch_dispatched_pending_jobsselects that row (Pendingwith a non-nulldispatch_time) andrecover_previous_jobsre-dispatches it throughrun(), which fails the same way. The row is recovered again on each restart and consumes a spawn each time.- Jobs that fail deserialization are already marked
Failed, so the extraUPDATEis wasted work.Move the push to the point where the handler is spawned.
🐛 Proposed fix
let job_id = job_row.id; - dispatched_job_ids.push(job_id); - let Some(clp_io_config) = self .try_deserialize_clp_io_config(job_id, &job_row.serialized_clp_io_config) .await else { continue; }; tracing::info!(job_id = % job_id, "Scheduling new job."); let Ok(job_handle) = self.create_job_handle(job_id, clp_io_config).await else { continue; }; + dispatched_job_ids.push(job_id); tokio::spawn(async move {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/compression-coordinator/src/coordination.rs` around lines 323 - 342, Move the dispatched_job_ids.push(job_id) call in the new-job scheduling loop until after create_job_handle succeeds and the handler is spawned. Do not record jobs that fail try_deserialize_clp_io_config or create_job_handle; preserve the existing continue behavior for those failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/compression-coordinator/src/coordination.rs`:
- Around line 158-161: Update the coordination loop so jobs spawned by
schedule_new_jobs are passed to mark_jobs_dispatched before the interruptible
polling wait, or are marked during cancellation shutdown before returning.
Ensure cancellation cannot bypass dispatch_time persistence, preventing
fetch_new_job_rows from selecting those jobs again after restart.
- Around line 534-540: Update the deserialization function’s rustdoc to state
that it returns the deserialized ClpIoConfig on success and None when
deserialization fails, and revise the failure description to use separate,
grammatically complete clauses instead of joining them with a comma.
---
Outside diff comments:
In `@components/compression-coordinator/src/coordination.rs`:
- Around line 46-49: Update the constructor documentation describing recovery to
include both categories handled by recover_previous_jobs: Running jobs with a
Spider job ID and Pending jobs with a populated dispatch_time. Preserve the
existing note that recovered jobs resume immediately without a concurrency
limit.
- Around line 323-342: Move the dispatched_job_ids.push(job_id) call in the
new-job scheduling loop until after create_job_handle succeeds and the handler
is spawned. Do not record jobs that fail try_deserialize_clp_io_config or
create_job_handle; preserve the existing continue behavior for those failures.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 9a5e4d29-81d2-47df-bd08-f80e303b3542
📒 Files selected for processing (1)
components/compression-coordinator/src/coordination.rs
| /// `dispatch_time` marks jobs that have already been dispatched by the current coordinator, | ||
| /// preventing them from being dispatched again before their handlers persist the `Running` | ||
| /// state. These updates are batched and applied after the polling interval to reduce | ||
| /// contention with the handlers' `Running` state updates. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
The documented dispatch marking does not happen on the cancellation path.
The docstring states that dispatch_time prevents jobs from being dispatched again before their handlers persist the Running state. The loop breaks on cancellation during the polling interval, so mark_jobs_dispatched never runs for the jobs that schedule_new_jobs just spawned in that iteration. After a restart, those rows still have dispatch_time IS NULL, so fetch_new_job_rows selects them again and the new coordinator dispatches them a second time.
Mark the jobs as dispatched before the interruptible wait, or mark them during shutdown before returning.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@components/compression-coordinator/src/coordination.rs` around lines 158 -
161, Update the coordination loop so jobs spawned by schedule_new_jobs are
passed to mark_jobs_dispatched before the interruptible polling wait, or are
marked during cancellation shutdown before returning. Ensure cancellation cannot
bypass dispatch_time persistence, preventing fetch_new_job_rows from selecting
those jobs again after restart.
There was a problem hiding this comment.
@LinZhihao-723 @sitaowang1998 This does seem like a real issue, although I think addressing it is outside the scope of this PR.
One possible solution would be to make claiming a job atomic by setting dispatch_time as part of the fetch operation. For example, fetch_new_job_rows could first claim a bounded set of pending jobs with:
UPDATE compression_jobs
SET dispatch_time = CURRENT_TIMESTAMP()
WHERE status = 'PENDING' AND dispatch_time IS NULL
ORDER BY id ASC
LIMIT ?Once a row is claimed, it can no longer be picked up as new work, regardless of whether the coordinator is cancelled or crashes between claiming and spawning the handler.
The remaining failure case is when a job is claimed but its handler never starts. In that case, the job remains Pending with dispatch_time set, so recover_previous_jobs can identify and re-dispatch it after restart.
There was a problem hiding this comment.
As explain https://github.com/y-scope/clp/pull/2435/changes#r3736798212: I think it should be fine if we fix the dispatch time cleaning.
dispatch_time is just a marker and it should be reset if spider_id is not set yet. It is possible to see cases where dispatch time is not set but it has a spider ID, which is fine and it can be properly handled by the recovery path.
There was a problem hiding this comment.
Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!
…lt since its error is unused
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/compression-coordinator/src/coordination.rs`:
- Around line 290-292: Update the documentation for the job-fetching/spawning
method around create_job_handle to state that UnsupportedInputConfig is logged
and left pending rather than marked failed, while retaining the failure
description for other handle-construction errors.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: d268beb8-e4d6-4e45-8161-c9f29e6f20c9
📒 Files selected for processing (1)
components/compression-coordinator/src/coordination.rs
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
components/compression-coordinator/src/coordination.rs (2)
322-334: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winAdd job IDs only after a handler is ready to spawn.
dispatched_job_idsis updated before deserialisation andcreate_job_handle. ForError::UnsupportedInputConfig,create_job_handlereturnsNoneand leaves the job pending, butmark_jobs_dispatchedstill sets itsdispatch_time. The coordinator then excludes the row from normal fetches and treats it as previously dispatched during recovery, although no handler exists. Move the push after successful handle creation.Proposed fix
let job_id = job_row.id; - dispatched_job_ids.push(job_id); let Some(clp_io_config) = self .try_deserialize_clp_io_config(job_id, &job_row.serialized_clp_io_config) .await @@ let Some(job_handle) = self.create_job_handle(job_id, clp_io_config).await else { continue; }; + dispatched_job_ids.push(job_id); tokio::spawn(async move {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/compression-coordinator/src/coordination.rs` around lines 322 - 334, Move the dispatched_job_ids.push(job_id) call in the job scheduling flow to immediately after create_job_handle successfully returns a handle, while retaining the existing deserialization and creation failure continues. Ensure only jobs with a ready-to-spawn handler are included when mark_jobs_dispatched processes the IDs.
492-520: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy liftBound the pending recovery result set.
fetch_dispatched_pending_jobsloads every pending row with a non-nulldispatch_timeinto oneVec.recover_previous_jobsthen spawns one detached task per row. A large recovery backlog can allocate excessive memory and create tasks independently ofmax_concurrent_tasks. Fetch recovery rows in bounded pages and route excess work through the same tracked admission path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/compression-coordinator/src/coordination.rs` around lines 492 - 520, The fetch_dispatched_pending_jobs and recover_previous_jobs flow must stop loading and spawning the entire recovery backlog at once. Change recovery-row retrieval to bounded pages, iterating until no rows remain, and submit each page’s jobs through the existing tracked admission mechanism governed by max_concurrent_tasks rather than detached per-row tasks; preserve the current pending-status and dispatch_time filtering.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@components/compression-coordinator/src/coordination.rs`:
- Around line 322-334: Move the dispatched_job_ids.push(job_id) call in the job
scheduling flow to immediately after create_job_handle successfully returns a
handle, while retaining the existing deserialization and creation failure
continues. Ensure only jobs with a ready-to-spawn handler are included when
mark_jobs_dispatched processes the IDs.
- Around line 492-520: The fetch_dispatched_pending_jobs and
recover_previous_jobs flow must stop loading and spawning the entire recovery
backlog at once. Change recovery-row retrieval to bounded pages, iterating until
no rows remain, and submit each page’s jobs through the existing tracked
admission mechanism governed by max_concurrent_tasks rather than detached
per-row tasks; preserve the current pending-status and dispatch_time filtering.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: b8b78f55-7627-4e58-b027-2c77782a6634
📒 Files selected for processing (1)
components/compression-coordinator/src/coordination.rs
LinZhihao-723
left a comment
LinZhihao-723
left a comment
There was a problem hiding this comment.
There's a risk that we won't be able to merge this PR before the release since it's not in an expected shape yet. Let's fix the recovery path bug first.
| async fn try_deserialize_clp_io_config( | ||
| &self, | ||
| job_id: CompressionJobId, | ||
| serialized_config: &[u8], | ||
| ) -> Option<ClpIoConfig> { | ||
| match BrotliMsgpack::deserialize(serialized_config) { | ||
| Ok(clp_io_config) => Some(clp_io_config), | ||
| Err(e) => { | ||
| tracing::error!( | ||
| error = % e, | ||
| job_id = % job_id, | ||
| "Failed to deserialize CLP I/O config. Skipping." | ||
| ); | ||
| self.mark_job_failed( | ||
| job_id, | ||
| &format!("Failed to deserialize CLP I/O config: {e}"), | ||
| ) | ||
| .await; | ||
| None | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
It's a really bad idea to return an option instead of a result, and log inside the helper. The best practice for designing this type of helper is:
- Return a result. Let the caller decide whether it wants to inspect the error or just propagate.
- Log on the caller side.
- You may see
create_job_handlelogs inside the helper. That is because it needs to print different logs based on the type of the error and execute different reaction calls accordingly. This makes it an exception since logging it inside would actually make things cleaner. In this helper, errors are printed unconditionally so it should be logged outside.
- You may see
| let Ok(job_handle) = self.create_job_handle(job_id, clp_io_config).await else { | ||
| let Some(job_handle) = self.create_job_handle(job_id, clp_io_config).await else { |
There was a problem hiding this comment.
Similar to https://github.com/y-scope/clp/pull/2435/changes#r3736535562: you didn't change the implementation of create_job_handle, then why would you need to update the return type? This is polluting the diff in some sense and making the PR harder to review.
| Some(id) => job_handle.recover(id).await, | ||
| None => job_handle.run().await, |
There was a problem hiding this comment.
I think this is actually a bug from my previous PR: at this stage, we should not handle any job that is already dispatched but doesn't have a spider ID.
The correct thing to do is (which is also what we actually expected), at the beginning of the factory, we should reset dispatch to NULL for all jobs that have a dispatch time set but no spider ID. These jobs should fall into the normal run loop instead of being handled in this recovery stage.
| /// A projection of the columns read from a compression job row. | ||
| #[derive(Debug, sqlx::FromRow)] | ||
| struct RunningJobRowProjection { | ||
| struct JobRowProjection { |
There was a problem hiding this comment.
It's a bad idea to merge PendingJobRowProjection and RunningJobRowProjection into one type: the user needs to check whether spider_job_id is NULL now. Since we will have https://github.com/y-scope/clp/pull/2435/changes#r3736798212, let roll back to two separate types instead of one.
In Rust, if you really want a type to cover both, a better idea is probably to make an enum containing both types.
| /// `dispatch_time` marks jobs that have already been dispatched by the current coordinator, | ||
| /// preventing them from being dispatched again before their handlers persist the `Running` | ||
| /// state. These updates are batched and applied after the polling interval to reduce | ||
| /// contention with the handlers' `Running` state updates. |
There was a problem hiding this comment.
As explain https://github.com/y-scope/clp/pull/2435/changes#r3736798212: I think it should be fine if we fix the dispatch time cleaning.
dispatch_time is just a marker and it should be reset if spider_id is not set yet. It is possible to see cases where dispatch time is not set but it has a spider ID, which is fine and it can be properly handled by the recovery path.
| continue; | ||
| }; | ||
|
|
||
| let permit = self.job_handler_sem.clone().try_acquire_owned().ok(); |
There was a problem hiding this comment.
I'm not sure if we really need to enforce the permit here: the recovery may not necessarily happen from a failure recovery, but a restart with clp config changed. If the new max_concurrent_jobs is changed to a smaller number, this recovery path will be blocked for an improper reason: eventually, a number of compression jobs are already submitted to Spider that exceeds the configured concurrency.
My proposal:
- For now, it's probably easier if we don't do anything to the permit: let the already-submitted jobs run in background. As we would assume the recovery is a rare case, the number of jobs should be roughly bounded by
old_max_concurrent_jobs+new_max_concurrent_jobs. There are extreme cases where this bound can be broken by a series of frequent restarts, but it should be safe to ignore this case for now. - To improve the naive implementation, we could compare the number of jobs to recover vs. the currently configured
max_concurrent_jobs:- If
num_jobs_to_recover>max_concurrent_jobs: print a warning, make a join of all the recovered job handlers, so the main loop only starts when all previously submitted jobs are finished. - Otherwise, do the current implementation to recover jobs while each job holds an acquired permit.
- If
| class CompressionCoordinator(BaseModel): | ||
| resource_group: SpiderResourceGroup = SpiderResourceGroup(name="compression-coordinator") | ||
| job_polling_interval_millisecs: PositiveInt = 100 | ||
| max_concurrent_tasks: PositiveInt = 1000 |
There was a problem hiding this comment.
The concurrency control is in the job level not the task level. Shouldn't it be called max_concurrent_jobs?
3 participants
Description
This PR limits the number of compression jobs processed concurrently to prevent the coordinator from creating an unbounded number of job-handler tasks and consuming excessive memory when a large backlog accumulates.
The coordinator now only fetches as many new jobs as it has capacity to process. Once the configured concurrency limit is reached, no additional jobs are fetched until existing jobs finish and capacity becomes available.
On startup, the coordinator recovers work from the previous instance before scheduling new jobs. This includes both jobs that were already submitted to Spider and jobs that were dispatched but did not progress far enough to be marked as running. The former resume tracking their existing Spider jobs, while the latter are re-dispatched.
NOTE
The
is_first_fetchflag and its two-query fetch logic have been removed because they do not work well with bounded concurrency. Previously, the first fetch after startup returned all pending jobs, including those with an existingdispatch_time, to recover jobs that may not have completed dispatch before the previous coordinator exited.With bounded concurrency, recovery can no longer rely on the first fetch returning everything, since the coordinator should only fetch as many new jobs as it has capacity to process. Recovery is therefore handled explicitly by
recover_previous_jobs, which identifies all pending jobs with an existingdispatch_timeand re-dispatches them before normal scheduling begins.This leaves the steady-state dispatcher with a single, clear eligibility rule: only pending jobs without a
dispatch_timeare considered new work.The concurrency limit is configured through a new
max_concurrent_tasksfield inClpConfig, with a default of 1000.Checklist
breaking change.
Validation performed
Summary by CodeRabbit
Summary by CodeRabbit
New Features
Bug Fixes