docs(compression-coordinator): Add a user guide for using compression-coordinator.

components/api-server/README.md

@@ -0,0 +1,11 @@
+# API Server
+
+The CLP API Server is a Rust-based HTTP server that exposes CLP functionality through REST APIs.
+
+## API reference
+
+All available API endpoints are defined in the [OpenAPI][openapi] Specification. The generated
+specification can be found [here][static-api-reference].
+
+[openapi]: https://swagger.io/specification/
+[static-api-reference]: ../../docs/src/_static/generated/api-server-openapi.json

components/clp-rust-utils/README.md

@@ -0,0 +1,3 @@
+# CLP Rust Utilities
+
+This Rust crate contains common utilities used by other Rust components in the CLP package.

components/clp-tdl-package/README.md

@@ -0,0 +1,14 @@
+# CLP TDL Package
+
+This component contains implementations of CLP tasks that run on Spider. It provides a shared
+library that conforms to Spider's Task Definition Language (TDL) package specification.
+
+## Tasks
+
+This component bundles all supported CLP tasks into a single TDL package. The available tasks are
+documented below.
+
+### Compression
+
+* `compression::clp_s_s3_compress`: Compress inputs from S3 using `clp-s`.
+* `compression::commit`: Commit compression task outcomes to the CLP metadata database.

components/compression-coordinator/README.md

@@ -0,0 +1,4 @@
+# Compression Coordinator
+
+This Rust component coordinates CLP compression jobs. It fetches newly created jobs, submits them to
+Spider, and monitors their progress.

components/log-ingestor/README.md

@@ -5,7 +5,6 @@ log ingestion jobs.
 
 ## Testing
 
-
 ### Quick Start
 
 All test cases in this component will be called through the Rust test task:

docs/src/dev-docs/design-deployment-orchestration.md

@@ -45,6 +45,7 @@ graph LR
   results_cache["results-cache (MongoDB)"]
   compression_scheduler["compression-scheduler"]
   query_scheduler["query-scheduler"]
+  compression_coordinator["compression-coordinator"]
   compression_worker["compression-worker"]
   query_worker["query-worker"]
   reducer["reducer"]
@@ -54,6 +55,9 @@ graph LR
   mcp_server["mcp-server"]
   log_ingestor["log-ingestor"]
 
+  %% Spider services
+  spider_service["spider-service"]
+
   %% One-time jobs
   db_table_creator["db-table-creator"]
   results_cache_indices_creator["results-cache-indices-creator"]
@@ -75,23 +79,28 @@ graph LR
   query_scheduler -->|healthy| reducer
   linkStyle 6 stroke:#800080
 
-  %% Link 7-13: Database initialization job --> Services
+  %% Link 7-14: Database initialization job --> Services
   db_table_creator -->|completed_successfully| api_server
+  db_table_creator -->|completed_successfully| compression_coordinator
   db_table_creator -->|completed_successfully| compression_scheduler
   db_table_creator -->|completed_successfully| garbage_collector
   db_table_creator -->|completed_successfully| log_ingestor
   db_table_creator -->|completed_successfully| mcp_server
   db_table_creator -->|completed_successfully| query_scheduler
   db_table_creator -->|completed_successfully| webui
-  linkStyle 7,8,9,10,11,12,13 stroke:#0000ff
+  linkStyle 7,8,9,10,11,12,13,14 stroke:#0000ff
 
-  %% Link 14-18: Results cache initialization job --> Services
+  %% Link 15-19: Results cache initialization job --> Services
   results_cache_indices_creator -->|completed_successfully| api_server
   results_cache_indices_creator -->|completed_successfully| garbage_collector
   results_cache_indices_creator -->|completed_successfully| mcp_server
   results_cache_indices_creator -->|completed_successfully| reducer
   results_cache_indices_creator -->|completed_successfully| webui
-  linkStyle 14,15,16,17,18 stroke:#008000
+  linkStyle 15,16,17,18,19 stroke:#008000
+
+  %% Link 20: Spider service --> Coordinators
+  spider_service -->|healthy| compression_coordinator
+  linkStyle 20 stroke:#008080
 
   subgraph Databases
     database
@@ -102,6 +111,10 @@ graph LR
     end
   end
 
+  subgraph Spider
+    spider_service
+  end
+
   subgraph Initialization jobs
     db_table_creator
     results_cache_indices_creator
@@ -112,6 +125,10 @@ graph LR
     query_scheduler
   end
 
+  subgraph Coordinators
+    compression_coordinator
+  end
+
   subgraph Workers
     compression_worker
     query_worker
@@ -142,22 +159,24 @@ graph LR
 :::{table}
 :align: left
 
-| Service                   | Description                                                        |
-|---------------------------|--------------------------------------------------------------------|
-| database                  | Database for archive metadata, compression jobs, and query jobs    |
-| queue                     | Task queue for schedulers                                          |
-| redis                     | Task result storage for workers                                    |
-| compression_scheduler     | Scheduler for compression jobs                                     |
-| query_scheduler           | Scheduler for search/aggregation jobs                              |
-| results_cache             | Storage for the workers to return search results to the UI         |
-| compression_worker        | Worker processes for compression jobs using Celery                 |
-| query_worker              | Worker processes for search/aggregation jobs using Celery          |
-| reducer                   | Reducers for performing the final stages of aggregation jobs       |
-| api_server                | API server for submitting queries                                  |
-| webui                     | Web server for the UI                                              |
-| mcp_server                | MCP server for AI agent to access CLP functionalities              |
-| garbage_collector         | Process to manage data retention                                   |
-| log_ingestor              | Server for orchestrating and running continuous log ingestion jobs |
+| Service                 | Description                                                                             |
+|-------------------------|-----------------------------------------------------------------------------------------|
+| database                | Database for archive metadata, compression jobs, and query jobs                         |
+| queue                   | Task queue for schedulers                                                               |
+| redis                   | Task result storage for workers                                                         |
+| compression_scheduler   | Scheduler for compression jobs                                                          |
+| query_scheduler         | Scheduler for search/aggregation jobs                                                   |
+| results_cache           | Storage for the workers to return search results to the UI                              |
+| compression_worker      | Worker processes for compression jobs using Celery                                      |
+| query_worker            | Worker processes for search/aggregation jobs using Celery                               |
+| reducer                 | Reducers for performing the final stages of aggregation jobs                            |
+| api_server              | API server for submitting queries                                                       |
+| webui                   | Web server for the UI                                                                   |
+| mcp_server              | MCP server for AI agent to access CLP functionalities                                   |
+| garbage_collector       | Process to manage data retention                                                        |
+| spider_service          | Spider distributed scheduling framework, containing sub-services such as Spider workers |
+| compression_coordinator | Coordinator for running CLP compression jobs on Spider                                  |
+| log_ingestor            | Server for orchestrating and running continuous log ingestion jobs                      |
 
 :::
 
@@ -241,15 +260,55 @@ Services require persistent storage for logs, data, archives, and streams.
 
 ### Deployment types
 
-CLP supports two deployment configurations, depending on the query engine:
+CLP supports four deployment configurations, determined by two independent choices: the query engine
+(Celery or Presto), and whether Spider is enabled.
+
+The query engine determines how queries are orchestrated:
 
 * **Full**: Uses Celery for query orchestration.
 * **Base**: Excludes the Celery-based query orchestration in Full, to support the
   [Presto integration][presto-integration].
 
+Enabling Spider adds the `compression-coordinator` service, which routes compression jobs to be
+driven by Spider instead of Celery. This choice is orthogonal to the query engine: queries are still
+driven by Celery or Presto according to the deployment type.
+
+:::{note}
+The Celery-based `compression-scheduler` will be deprecated in a future release to be fully replaced
+by `compression-coordinator` and Spider. However, it still remains enabled by default since the
+current `compression-coordinator` implementation can only coordinate compression jobs created by
+`log-ingestor`.
+:::
+
+[Table 3](#table-3) below lists the resulting deployment configurations.
+
+(table-3)=
+::::{card}
+
+:::{table}
+:align: left
+
+| Deployment type | Spider   | Query orchestration | Compression orchestration                                  |
+|-----------------|----------|---------------------|------------------------------------------------------------|
+| Full            | Disabled | Celery              | Celery                                                     |
+| Base            | Disabled | Presto              | Celery                                                     |
+| Full            | Enabled  | Celery              | Spider for jobs from `log-ingestor`; Celery for all others |
+| Base            | Enabled  | Presto              | Spider for jobs from `log-ingestor`; Celery for all others |
+
+:::
+
++++
+**Table 3**: Deployment configurations in the CLP package.
+::::
+
 Docker Compose uses `deploy.replicas` with environment variables (e.g., `CLP_MCP_SERVER_ENABLED`)
 to toggle optional services. Helm uses conditional templating to include or exclude resources.
 
+:::{note}
+Spider is only available for Helm deployments in the current CLP package. Docker Compose deployments
+cannot enable Spider.
+:::
+
 ## Troubleshooting
 
 When issues arise, use the appropriate commands for your orchestration method:

docs/src/dev-docs/design-project-structure.md

@@ -2,20 +2,30 @@
 
 CLP is currently split across a few different components in the [components] directory:
 
-* [clp-mcp-server] contains code for CLP MCP Server.
+* [api-server] contains code for the API server of the CLP package.
+* [clp-mcp-server] contains code for CLP MCP server.
 * [clp-package-utils] contains Python utilities for operating the CLP package.
-* [clp-py-utils] contains Python utilities common to several of the other components.
+* [clp-py-utils] contains Python utilities common to several of the other Python components.
+* [clp-rust-utils] contains Rust utilities common to several of the other Rust components.
+* [clp-tdl-package] contains implementations of CLP tasks running on Spider.
+* [compression-coordinator] contains code to coordinate CLP compression jobs running on Spider.
 * [core] contains code to compress uncompressed logs, decompress compressed logs, and search
   compressed logs.
-* [job-orchestration] contains code to schedule compression jobs on the cluster.
+* [job-orchestration] contains code to schedule compression and search jobs on the cluster.
+* [log-ingestor] contains code to serve requests for CLP ingestion job orchestration.
 * [package-template] contains the base directory structure and files of the CLP package.
 * [webui] contains the web interface for the CLP package.
 
+[api-server]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/api-server
 [components]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components
 [clp-mcp-server]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-mcp-server
 [clp-package-utils]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-package-utils
 [clp-py-utils]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-py-utils
+[clp-rust-utils]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-rust-utils
+[clp-tdl-package]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-tdl-package
+[compression-coordinator]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/compression-coordinator
 [core]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/core
 [job-orchestration]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/job-orchestration
+[log-ingestor]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/log-ingestor
 [package-template]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/package-template
 [webui]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/webui

docs/src/user-docs/guides-using-compression-coordinator.md

@@ -0,0 +1,60 @@
+# Using `compression-coordinator`
+
+`compression-coordinator` is a new component that coordinates compression jobs on [Spider][spider].
+It is designed to replace the existing `compression-scheduler`, which schedules compression jobs
+using Celery.
+
+To use `compression-coordinator`, enable the Spider scheduling framework when starting the CLP
+package.
+
+TODO: Link to Chenxing's doc.
+
+:::{note}
+Currently, `compression-coordinator` can be deployed only on Kubernetes using the CLP Helm chart.
+Support for Docker Compose deployments is planned for a future release.
+:::
+
+Compared with `compression-scheduler`, `compression-coordinator` provides the following improvements
+to the user experience, reliability, and performance:
+
+* **Automatic failure recovery**: If a service fails or restarts, `compression-coordinator`
+  automatically resumes previously submitted jobs. Users do not need to restart them manually.
+* **Improved resource utilization**: Compression jobs run in a Spider-managed execution environment,
+  allowing computational resources to be shared more effectively across all configured workers.
+* **Configurable retries for compression failures**: `compression-coordinator` allows users to
+  configure a simple retry policy for failed compression tasks. Automatic retries can help recover
+  from transient issues, such as temporary network interruptions, without requiring user
+  intervention.
+  * TODO: Configure retry policy through <LINK>
+* **Improved fairness across concurrent compression jobs**: `compression-coordinator` provides
+  two levels of concurrency control to improve fairness among compression jobs running concurrently:
+  * Coordinator-side rate limit: TODO, depends on #2435.
+  * Spider-side active job limit: Controls the maximum number of jobs that can make progress
+    concurrently while sharing compute resources in Spider. Jobs are admitted on a first-come,
+    first-served basis.
+* **Data integrity**: Compression jobs coordinated by `compression-coordinator` publish their
+  results to the rest of the system through a dedicated commit stage. The commit operation is both
+  transactional and idempotent.
+  * A job-level failure does not result in partial updates.
+  * Internal retries do not result in duplicate updates
+
+:::{note}
+In this release, `compression-coordinator` has the following functional limitations:
+
+* It supports only the `clp-json` package.
+* It handles only compression jobs created by `log-ingestor`.
+* It supports only S3 archive output.
+
+We are actively expanding the functionality of `compression-coordinator`. Additional capabilities
+will be introduced in future releases as it moves toward feature parity with
+`compression-scheduler`.
+:::
+
+:::{note}
+`compression-scheduler` is planned for deprecation and will eventually be fully replaced by
+`compression-coordinator`. In this release, however, `compression-scheduler` continues to run
+alongside `compression-coordinator` to handle compression jobs that `compression-coordinator` does
+not yet support.
+:::
+
+[spider]: https://github.com/y-scope/spider

docs/src/user-docs/index.md

@@ -82,6 +82,13 @@ Configuring retention periods
 ^^^
 Configure retention periods for archives and search results.
 :::
+
+:::{grid-item-card}
+:link: guides-compression-coordinator
+Using `compression-coordinator`
+^^^
+Using `compression-coordinator` to coordinate compression jobs.
+:::
 ::::
 
 ---
@@ -248,6 +255,7 @@ guides-using-object-storage/index
 guides-using-log-ingestor
 guides-external-database
 guides-retention
+guides-using-compression-coordinator
 :::
 
 :::{toctree}