feat(helm)!: Add the Spider subchart and compression-coordinator wiring for Spider-orchestrated compression.

[20001020ycx]

Description

See PR title. At a high level, this PR integrates Spider into the CLP Helm chart as an optional subchart that, when enabled, orchestrates compression jobs in place of the Celery-based scheduler and workers. Note, compression coordinator, compression scheduler, and compression worker are on regardless of Spider's toggle; their internal implementation handles the routing of the jobs with respect to the types.

Note: the scope of the integration is only limited to compression jobs, search jobs routing shall fall back to the Celery's query scheduler + worker.

Important

Blocked on y-scope/spider#413，y-scope/spider#415, #2408, complete compression coordinator implementation (#2417, #2420, and #2421 ), and complete TDL package implementation (#2406, #2411, #2415).

Note

This PR is breaking because we set fullnameOverride to global. A user now needs to run helm install --set global.fullnameOverride rather than helm install --set fullnameOverride.

Validation performed

Note

The validation below uses only official artifacts:

• ghcr.io/y-scope/clp/clp-package and ghcr.io/y-scope/clp/clp-spider-worker, CI-published from main
• the Spider subchart (0.1.8) resolved from the Spider repo's gh-pages
• an in-cluster MinIO as the S3 endpoint (via s3_config.endpoint_url), standing in for AWS S3

Full-Spider compression E2E: the job executes in Spider and the result is searchable

0. Environment: a single-node kind cluster; an S3 bucket clp-logs with 4 JSONL log objects under demo-logs/, e.g. {"ts":1700000001,"level":"INFO","msg":"demo log line 1"}; an empty S3 bucket clp-archives for archive output.

1. Overlay runtime values for values.yaml during deployment, you may review the complete values.yaml here.

spider:
  enabled: true

clpConfig:
  logs_input: # S3 ingestion config (type s3 + bucket credentials).
  archive_output: # S3 archive-output config; `region_code` must be omitted with a custom endpoint_url (the query path would build http://<region>.<endpoint-host>/..., which doesn't resolve).

helm dependency update tools/deployment/package-helm
helm install test tools/deployment/package-helm -f <overlay above>
kubectl get pods | grep -E "compression|ingestor|spider"

Expected (no compression-scheduler/compression-worker pods exist):

spider-database-0                                   1/1     Running
spider-scheduler-95fd9ddf-s29qm                     1/1     Running
spider-storage-76fd7d95b5-dtwp8                     1/1     Running
spider-worker-6996fd4b8f-5kzm9                      1/1     Running
test-clp-compression-coordinator-5765bbb96b-vssl5   1/1     Running
test-clp-log-ingestor-7bbfd478d7-k9dmk              1/1     Running

2. Start ingestion through the log-ingestor's REST API:

curl -X POST http://test-clp-log-ingestor:3002/s3_scanner -H 'Content-Type: application/json' \
  -d '{"bucket_name": "clp-logs", "key_prefix": "demo-logs/", "endpoint_url": "http://minio:9000",
       "region": "us-east-1", "scanning_interval_sec": 5, "buffer_config": {"timeout_sec": 10}}'

Expected: {"id":1}. ~30 s later, in the CLP DB:

kubectl exec test-clp-database-0 -- mariadb --table -u clp-user -ppass clp-db \
  -e "SELECT id, \`key\`, status, compression_job_id FROM ingested_s3_object_metadata;"

id	key	status	compression_job_id
1	demo-logs/app-1.jsonl	compressed	1
2	demo-logs/app-2.jsonl	compressed	1
3	demo-logs/app-3.jsonl	compressed	1
4	demo-logs/app-4.jsonl	compressed	1

3. The compression coordinator drove the full submit→execute→result loop:

kubectl logs deploy/test-clp-compression-coordinator | grep -oE '"message":"[^"]*"|"outcome":"[^"]*"'

Expected:

"message":"Scheduling new job."
"message":"Starting compression job."
"message":"Submitted compression job to Spider."
"message":"Compression job submitted."
"message":"Compression job submission persisted."
"message":"Compression job reached a terminal state."
"outcome":"Succeeded"

and in the CLP DB (status=2 is SUCCEEDED):

kubectl exec test-clp-database-0 -- mariadb --table -u clp-user -ppass clp-db \
  -e "SELECT id, status, num_tasks, spider_id FROM compression_jobs;"

id	status	num_tasks	spider_id
1	2	1	1

4. Spider's own database confirms the execution:

kubectl exec spider-database-0 -- mariadb --table -u spider-user -pspider-password spider-db \
  -e "SELECT id, resource_group_id, state FROM jobs; SELECT external_id FROM resource_groups;
      SELECT id, state FROM execution_managers;"

Expected:

jobs

id	resource_group_id	state
1	1	Succeeded

resource_groups

external_id
compression-coordinator

execution_managers

id	state
1	Alive

5. The archive the job produced is traceable end to end, at spider's job table:

kubectl exec spider-database-0 -- mariadb -u spider-user -pspider-password spider-db \
  -e "SELECT serialized_job_outputs FROM jobs\G"

serialized_job_outputs:  ...default...$8ae188fb-8775-414a-8bdb-9b135ec91292...

The commit task registered the same archive in CLP's metadata table:

kubectl exec test-clp-database-0 -- mariadb --table -u clp-user -ppass clp-db \
  -e "SELECT id, uncompressed_size, size FROM clp_default_archives;"

id	uncompressed_size	size
8ae188fb-8775-414a-8bdb-9b135ec91292	228	648

The compress task uploaded the same archive to the output bucket:

mc ls -r s3/clp-archives/

[2026-07-24 21:08:22 UTC]   648B STANDARD archives/default/8ae188fb-8775-414a-8bdb-9b135ec91292

6. Search returns the original log events from the Spider-compressed archive; KQL: msg: "demo log line*":

curl -X POST http://test-clp-api-server:3001/query -H 'Content-Type: application/json' \
  -d '{"query_string": "msg: \"demo log line*\"", "datasets": ["default"], "ignore_case": true,
       "max_num_results": 10, "buffer_results_in_mongodb": true}'   # -> {"query_results_uri":"/query_results/1"}
curl http://test-clp-api-server:3001/query_results/1

Expected:

data: {"ts":1700000001,"level":"INFO","msg":"demo log line 1"}
data: {"ts":1700000002,"level":"INFO","msg":"demo log line 2"}
data: {"ts":1700000003,"level":"INFO","msg":"demo log line 3"}
data: {"ts":1700000004,"level":"INFO","msg":"demo log line 4"}

Summary by CodeRabbit

Summary by CodeRabbit

• New Features

  • Added optional Spider storage integration for Helm deployments, including storage host/port helpers.
  • Introduced a configurable compression coordinator component.

• Bug Fixes

  • Improved Helm packaging and test install flows by updating chart dependencies immediately before packaging/installation.
  • Enhanced Helm linting to resolve the Spider chart repository during lint checks.

• Chores

  • Bumped the Helm chart version, wired Spider chart dependency, and excluded downloaded subcharts from source control.
  • Updated Helm templates/config so the coordinator renders when Spider is enabled, and related compression components are omitted accordingly.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

The Helm chart adds a Spider dependency, resolves it during packaging and test setup, introduces Spider-backed coordinator configuration, and conditionally replaces compression scheduler and worker deployments with a compression coordinator deployment when Spider mode is enabled.

Changes

Spider Helm integration

Layer / File(s)	Summary
Chart dependency and packaging flow taskfiles/helm.yaml, taskfiles/lint.yaml, tools/deployment/package-helm/Chart.yaml, tools/deployment/package-helm/set-up-*.sh, tools/deployment/package-helm/.gitignore	The chart declares Spider version 0.1.4, builds dependencies before packaging and installation, registers the Spider repository for linting, and ignores generated chart dependencies.
Spider configuration and coordinator settings tools/deployment/package-helm/values.yaml, tools/deployment/package-helm/templates/_helpers.tpl, tools/deployment/package-helm/templates/configmap.yaml	Values and helpers define coordinator settings and Spider storage addressing, while the ConfigMap renders them when Spider is enabled.
Conditional compression deployments tools/deployment/package-helm/templates/compression-coordinator-deployment.yaml, tools/deployment/package-helm/templates/compression-scheduler-deployment.yaml, tools/deployment/package-helm/templates/compression-worker-deployment.yaml	A Spider-gated compression coordinator Deployment is added, and the existing scheduler and worker Deployments are omitted in Spider mode.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Helm
  participant Kubernetes
  participant DbTableCreator
  participant StorageService
  Helm->>Kubernetes: Render compression-coordinator Deployment
  Kubernetes->>DbTableCreator: Wait for db-table-creator job
  Kubernetes->>StorageService: Wait for storage service
  Kubernetes->>Kubernetes: Start compression-coordinator

Loading

Suggested reviewers: junhaoliao, hoophalab

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the Spider subchart integration and compression-coordinator wiring, which are the main changes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@tools/deployment/package-helm/templates/compression-coordinator-deployment.yaml`:
- Around line 27-37: Update the service wait target in the compression
coordinator’s initContainers to use the Spider storage service name, resolving
to “spider-storage” via the existing fullname override, instead of the component
name “storage”; leave the db-table-creator job wait unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 3ee44ae5-21c4-4843-808b-d682a7ff3558

📥 Commits

Reviewing files that changed from the base of the PR and between d3c7d45 and f4f2b83.

⛔ Files ignored due to path filters (1)

• tools/deployment/package-helm/Chart.lock is excluded by !**/*.lock

📒 Files selected for processing (13)

• taskfiles/helm.yaml
• taskfiles/lint.yaml
• tools/deployment/package-helm/.gitignore
• tools/deployment/package-helm/Chart.yaml
• tools/deployment/package-helm/set-up-multi-dedicated-test.sh
• tools/deployment/package-helm/set-up-multi-shared-test.sh
• tools/deployment/package-helm/set-up-test.sh
• tools/deployment/package-helm/templates/_helpers.tpl
• tools/deployment/package-helm/templates/compression-coordinator-deployment.yaml
• tools/deployment/package-helm/templates/compression-scheduler-deployment.yaml
• tools/deployment/package-helm/templates/compression-worker-deployment.yaml
• tools/deployment/package-helm/templates/configmap.yaml
• tools/deployment/package-helm/values.yaml

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/deployment/package-helm/set-up-multi-dedicated-test.sh`:
- Line 141: Replace unconditional helm dependency update in
tools/deployment/package-helm/set-up-multi-dedicated-test.sh:141,
tools/deployment/package-helm/set-up-multi-shared-test.sh:47, and
tools/deployment/package-helm/set-up-test.sh:31 with dependency handling that
does not require the Spider repository when spider.enabled=false; either vendor
the Spider chart alongside each chart or make Spider acquisition explicit and
conditional on its availability or enabled state.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2c871999-cb32-4c01-97cf-08a373853f6d

📥 Commits

Reviewing files that changed from the base of the PR and between 309cefa and 7dfe382.

📒 Files selected for processing (3)

• tools/deployment/package-helm/set-up-multi-dedicated-test.sh
• tools/deployment/package-helm/set-up-multi-shared-test.sh
• tools/deployment/package-helm/set-up-test.sh

[hoophalab]

LGTM

Validation: the charts are deployed successfully.

[20001020ycx]

We have deployed the current state on the helm chart on the staging namespace of YScope Cloud; all pods are functioning, compression and search jobs complete successfully, provided that the credential-fix PR lands (#2438).

Greatly appreciate for the help from @hoophalab for the Cloud deployment testing.

[junhaoliao]

the rest lgtm. everything looks quite clean and the changes are idiomatic

[junhaoliao]

mind quickly explaining why we have to move those two keys to global? do we need this to also override spider's fullnameOverride and nameOverride (but spider's Helm chart would need modifications in the first place?)

[20001020ycx]

This is a great question which had haunted me for a while, and I am happy to explain as this is the problem with subchart.

Consider the below config we must set at Spider side such that worker can talk to CLP database (for commit task).

- name: "CLP_DB_USER"
     valueFrom:
       secretKeyRef:
       name: "{{ include \"clp.fullname\" . }}-database"

However, clp.fullname is only recognized by CLP's helm chart. Without declaring it as global, Spider subchart resolves it to the literal string {{ include \"clp.fullname\" . }} rather the actual full name of the clp subchart.

[junhaoliao]

I see

shall we mark this as a breaking change? as in, those who need to override those values should be aware they need to change different keys now

[20001020ycx]

I also performed regression test to ensure the old compression path using Celery does not fail. Below steps are performed:

0. Set up kind context

CTX="--context kind-clp-test"

1. Log file on the node

docker exec clp-test-control-plane sh -c \
  'mkdir -p /demo-logs && printf "{\"ts\":1700000011,\"level\":\"INFO\",\"msg\":\"webui fs line 11\"}\n" > /demo-logs/app.jsonl'

2. Install with defaults (fs mode) — from tools/deployment/package-helm

helm $CTX install test . \
  --set clpConfig.logs_input.directory=/demo-logs \
  --set image.clpPackage.tag=main-amd64 --set image.clpPackage.pullPolicy=IfNotPresent

3. Compress via webui (or use the Ingest page: kubectl port-forward svc/test-clp-webui 4000:4000)

kubectl $CTX run curl-compress --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- \
  -sS -X POST http://test-clp-webui:4000/api/compress \
  -H 'Content-Type: application/json' -d '{"paths": ["/app.jsonl"], "dataset": "default"}'
-> {"jobId":1}

4. Verify: Celery took it (spider_id NULL)

DBPASS=$(kubectl $CTX get secret test-clp-database -o jsonpath='{.data.password}' | base64 -d)
kubectl $CTX exec test-clp-database-0 -- mariadb --table -u clp-user -p$DBPASS clp-db \
  -e "SELECT id, status, spider_id, duration FROM compression_jobs;"
# -> id=1, status=2, spider_id=NULL

[junhaoliao]

lgtm. the title lgtm too but let's consider marking this as a breaking change before merging