diff --git a/components/api-server/README.md b/components/api-server/README.md new file mode 100644 index 000000000..de822c765 --- /dev/null +++ b/components/api-server/README.md @@ -0,0 +1,11 @@ +# API Server + +The CLP API Server is a Rust-based HTTP server that exposes CLP functionality through REST APIs. + +## API reference + +All available API endpoints are defined in the [OpenAPI][openapi] Specification. The generated +specification can be found [here][static-api-reference]. + +[openapi]: https://swagger.io/specification/ +[static-api-reference]: ../../docs/src/_static/generated/api-server-openapi.json diff --git a/components/clp-rust-utils/README.md b/components/clp-rust-utils/README.md new file mode 100644 index 000000000..6b8fb6a82 --- /dev/null +++ b/components/clp-rust-utils/README.md @@ -0,0 +1,3 @@ +# CLP Rust Utilities + +This Rust crate contains common utilities used by other Rust components in the CLP package. diff --git a/components/clp-tdl-package/README.md b/components/clp-tdl-package/README.md new file mode 100644 index 000000000..8e2cad700 --- /dev/null +++ b/components/clp-tdl-package/README.md @@ -0,0 +1,14 @@ +# CLP TDL Package + +This component contains implementations of CLP tasks that run on Spider. It provides a shared +library that conforms to Spider's Task Definition Language (TDL) package specification. + +## Tasks + +This component bundles all supported CLP tasks into a single TDL package. The available tasks are +documented below. + +### Compression + +* `compression::clp_s_s3_compress`: Compress inputs from S3 using `clp-s`. +* `compression::commit`: Commit compression task outcomes to the CLP metadata database. diff --git a/components/compression-coordinator/README.md b/components/compression-coordinator/README.md new file mode 100644 index 000000000..f7e80456e --- /dev/null +++ b/components/compression-coordinator/README.md @@ -0,0 +1,4 @@ +# Compression Coordinator + +This Rust component coordinates CLP compression jobs. It fetches newly created jobs, submits them to +Spider, and monitors their progress. diff --git a/components/log-ingestor/README.md b/components/log-ingestor/README.md index 8d7c9c8be..9064650f6 100644 --- a/components/log-ingestor/README.md +++ b/components/log-ingestor/README.md @@ -5,7 +5,6 @@ log ingestion jobs. ## Testing - ### Quick Start All test cases in this component will be called through the Rust test task: diff --git a/docs/src/dev-docs/design-deployment-orchestration.md b/docs/src/dev-docs/design-deployment-orchestration.md index 86b1e64ed..fc3543a7b 100755 --- a/docs/src/dev-docs/design-deployment-orchestration.md +++ b/docs/src/dev-docs/design-deployment-orchestration.md @@ -45,6 +45,7 @@ graph LR results_cache["results-cache (MongoDB)"] compression_scheduler["compression-scheduler"] query_scheduler["query-scheduler"] + compression_coordinator["compression-coordinator"] compression_worker["compression-worker"] query_worker["query-worker"] reducer["reducer"] @@ -54,6 +55,9 @@ graph LR mcp_server["mcp-server"] log_ingestor["log-ingestor"] + %% Spider services + spider_service["spider-service"] + %% One-time jobs db_table_creator["db-table-creator"] results_cache_indices_creator["results-cache-indices-creator"] @@ -75,23 +79,28 @@ graph LR query_scheduler -->|healthy| reducer linkStyle 6 stroke:#800080 - %% Link 7-13: Database initialization job --> Services + %% Link 7-14: Database initialization job --> Services db_table_creator -->|completed_successfully| api_server + db_table_creator -->|completed_successfully| compression_coordinator db_table_creator -->|completed_successfully| compression_scheduler db_table_creator -->|completed_successfully| garbage_collector db_table_creator -->|completed_successfully| log_ingestor db_table_creator -->|completed_successfully| mcp_server db_table_creator -->|completed_successfully| query_scheduler db_table_creator -->|completed_successfully| webui - linkStyle 7,8,9,10,11,12,13 stroke:#0000ff + linkStyle 7,8,9,10,11,12,13,14 stroke:#0000ff - %% Link 14-18: Results cache initialization job --> Services + %% Link 15-19: Results cache initialization job --> Services results_cache_indices_creator -->|completed_successfully| api_server results_cache_indices_creator -->|completed_successfully| garbage_collector results_cache_indices_creator -->|completed_successfully| mcp_server results_cache_indices_creator -->|completed_successfully| reducer results_cache_indices_creator -->|completed_successfully| webui - linkStyle 14,15,16,17,18 stroke:#008000 + linkStyle 15,16,17,18,19 stroke:#008000 + + %% Link 20: Spider service --> Coordinators + spider_service -->|healthy| compression_coordinator + linkStyle 20 stroke:#008080 subgraph Databases database @@ -102,6 +111,10 @@ graph LR end end + subgraph Spider + spider_service + end + subgraph Initialization jobs db_table_creator results_cache_indices_creator @@ -112,6 +125,10 @@ graph LR query_scheduler end + subgraph Coordinators + compression_coordinator + end + subgraph Workers compression_worker query_worker @@ -142,22 +159,24 @@ graph LR :::{table} :align: left -| Service | Description | -|---------------------------|--------------------------------------------------------------------| -| database | Database for archive metadata, compression jobs, and query jobs | -| queue | Task queue for schedulers | -| redis | Task result storage for workers | -| compression_scheduler | Scheduler for compression jobs | -| query_scheduler | Scheduler for search/aggregation jobs | -| results_cache | Storage for the workers to return search results to the UI | -| compression_worker | Worker processes for compression jobs using Celery | -| query_worker | Worker processes for search/aggregation jobs using Celery | -| reducer | Reducers for performing the final stages of aggregation jobs | -| api_server | API server for submitting queries | -| webui | Web server for the UI | -| mcp_server | MCP server for AI agent to access CLP functionalities | -| garbage_collector | Process to manage data retention | -| log_ingestor | Server for orchestrating and running continuous log ingestion jobs | +| Service | Description | +|-------------------------|-----------------------------------------------------------------------------------------| +| database | Database for archive metadata, compression jobs, and query jobs | +| queue | Task queue for schedulers | +| redis | Task result storage for workers | +| compression_scheduler | Scheduler for compression jobs | +| query_scheduler | Scheduler for search/aggregation jobs | +| results_cache | Storage for the workers to return search results to the UI | +| compression_worker | Worker processes for compression jobs using Celery | +| query_worker | Worker processes for search/aggregation jobs using Celery | +| reducer | Reducers for performing the final stages of aggregation jobs | +| api_server | API server for submitting queries | +| webui | Web server for the UI | +| mcp_server | MCP server for AI agent to access CLP functionalities | +| garbage_collector | Process to manage data retention | +| spider_service | Spider distributed scheduling framework, containing sub-services such as Spider workers | +| compression_coordinator | Coordinator for running CLP compression jobs on Spider | +| log_ingestor | Server for orchestrating and running continuous log ingestion jobs | ::: @@ -241,15 +260,55 @@ Services require persistent storage for logs, data, archives, and streams. ### Deployment types -CLP supports two deployment configurations, depending on the query engine: +CLP supports four deployment configurations, determined by two independent choices: the query engine +(Celery or Presto), and whether Spider is enabled. + +The query engine determines how queries are orchestrated: * **Full**: Uses Celery for query orchestration. * **Base**: Excludes the Celery-based query orchestration in Full, to support the [Presto integration][presto-integration]. +Enabling Spider adds the `compression-coordinator` service, which routes compression jobs to be +driven by Spider instead of Celery. This choice is orthogonal to the query engine: queries are still +driven by Celery or Presto according to the deployment type. + +:::{note} +The Celery-based `compression-scheduler` will be deprecated in a future release to be fully replaced +by `compression-coordinator` and Spider. However, it still remains enabled by default since the +current `compression-coordinator` implementation can only coordinate compression jobs created by +`log-ingestor`. +::: + +[Table 3](#table-3) below lists the resulting deployment configurations. + +(table-3)= +::::{card} + +:::{table} +:align: left + +| Deployment type | Spider | Query orchestration | Compression orchestration | +|-----------------|----------|---------------------|------------------------------------------------------------| +| Full | Disabled | Celery | Celery | +| Base | Disabled | Presto | Celery | +| Full | Enabled | Celery | Spider for jobs from `log-ingestor`; Celery for all others | +| Base | Enabled | Presto | Spider for jobs from `log-ingestor`; Celery for all others | + +::: + ++++ +**Table 3**: Deployment configurations in the CLP package. +:::: + Docker Compose uses `deploy.replicas` with environment variables (e.g., `CLP_MCP_SERVER_ENABLED`) to toggle optional services. Helm uses conditional templating to include or exclude resources. +:::{note} +Spider is only available for Helm deployments in the current CLP package. Docker Compose deployments +cannot enable Spider. +::: + ## Troubleshooting When issues arise, use the appropriate commands for your orchestration method: diff --git a/docs/src/dev-docs/design-project-structure.md b/docs/src/dev-docs/design-project-structure.md index f82c93410..edaeed779 100644 --- a/docs/src/dev-docs/design-project-structure.md +++ b/docs/src/dev-docs/design-project-structure.md @@ -2,20 +2,30 @@ CLP is currently split across a few different components in the [components] directory: -* [clp-mcp-server] contains code for CLP MCP Server. +* [api-server] contains code for the API server of the CLP package. +* [clp-mcp-server] contains code for CLP MCP server. * [clp-package-utils] contains Python utilities for operating the CLP package. -* [clp-py-utils] contains Python utilities common to several of the other components. +* [clp-py-utils] contains Python utilities common to several of the other Python components. +* [clp-rust-utils] contains Rust utilities common to several of the other Rust components. +* [clp-tdl-package] contains implementations of CLP tasks running on Spider. +* [compression-coordinator] contains code to coordinate CLP compression jobs running on Spider. * [core] contains code to compress uncompressed logs, decompress compressed logs, and search compressed logs. -* [job-orchestration] contains code to schedule compression jobs on the cluster. +* [job-orchestration] contains code to schedule compression and search jobs on the cluster. +* [log-ingestor] contains code to serve requests for CLP ingestion job orchestration. * [package-template] contains the base directory structure and files of the CLP package. * [webui] contains the web interface for the CLP package. +[api-server]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/api-server [components]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components [clp-mcp-server]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-mcp-server [clp-package-utils]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-package-utils [clp-py-utils]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-py-utils +[clp-rust-utils]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-rust-utils +[clp-tdl-package]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/clp-tdl-package +[compression-coordinator]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/compression-coordinator [core]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/core [job-orchestration]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/job-orchestration +[log-ingestor]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/log-ingestor [package-template]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/package-template [webui]: https://github.com/y-scope/clp/tree/DOCS_VAR_CLP_GIT_REF/components/webui diff --git a/docs/src/user-docs/guides-using-compression-coordinator.md b/docs/src/user-docs/guides-using-compression-coordinator.md new file mode 100644 index 000000000..fa270b57e --- /dev/null +++ b/docs/src/user-docs/guides-using-compression-coordinator.md @@ -0,0 +1,60 @@ +# Using `compression-coordinator` + +`compression-coordinator` is a new component that coordinates compression jobs on [Spider][spider]. +It is designed to replace the existing `compression-scheduler`, which schedules compression jobs +using Celery. + +To use `compression-coordinator`, enable the Spider scheduling framework when starting the CLP +package. + +TODO: Link to Chenxing's doc. + +:::{note} +Currently, `compression-coordinator` can be deployed only on Kubernetes using the CLP Helm chart. +Support for Docker Compose deployments is planned for a future release. +::: + +Compared with `compression-scheduler`, `compression-coordinator` provides the following improvements +to the user experience, reliability, and performance: + +* **Automatic failure recovery**: If a service fails or restarts, `compression-coordinator` + automatically resumes previously submitted jobs. Users do not need to restart them manually. +* **Improved resource utilization**: Compression jobs run in a Spider-managed execution environment, + allowing computational resources to be shared more effectively across all configured workers. +* **Configurable retries for compression failures**: `compression-coordinator` allows users to + configure a simple retry policy for failed compression tasks. Automatic retries can help recover + from transient issues, such as temporary network interruptions, without requiring user + intervention. + * TODO: Configure retry policy through +* **Improved fairness across concurrent compression jobs**: `compression-coordinator` provides + two levels of concurrency control to improve fairness among compression jobs running concurrently: + * Coordinator-side rate limit: TODO, depends on #2435. + * Spider-side active job limit: Controls the maximum number of jobs that can make progress + concurrently while sharing compute resources in Spider. Jobs are admitted on a first-come, + first-served basis. +* **Data integrity**: Compression jobs coordinated by `compression-coordinator` publish their + results to the rest of the system through a dedicated commit stage. The commit operation is both + transactional and idempotent. + * A job-level failure does not result in partial updates. + * Internal retries do not result in duplicate updates + +:::{note} +In this release, `compression-coordinator` has the following functional limitations: + +* It supports only the `clp-json` package. +* It handles only compression jobs created by `log-ingestor`. +* It supports only S3 archive output. + +We are actively expanding the functionality of `compression-coordinator`. Additional capabilities +will be introduced in future releases as it moves toward feature parity with +`compression-scheduler`. +::: + +:::{note} +`compression-scheduler` is planned for deprecation and will eventually be fully replaced by +`compression-coordinator`. In this release, however, `compression-scheduler` continues to run +alongside `compression-coordinator` to handle compression jobs that `compression-coordinator` does +not yet support. +::: + +[spider]: https://github.com/y-scope/spider diff --git a/docs/src/user-docs/index.md b/docs/src/user-docs/index.md index f10656eaf..b97ca46e9 100644 --- a/docs/src/user-docs/index.md +++ b/docs/src/user-docs/index.md @@ -82,6 +82,13 @@ Configuring retention periods ^^^ Configure retention periods for archives and search results. ::: + +:::{grid-item-card} +:link: guides-compression-coordinator +Using `compression-coordinator` +^^^ +Using `compression-coordinator` to coordinate compression jobs. +::: :::: --- @@ -248,6 +255,7 @@ guides-using-object-storage/index guides-using-log-ingestor guides-external-database guides-retention +guides-using-compression-coordinator ::: :::{toctree}