feat(deployment): Deploy Presto with stock images and install the CLP connector as a plugin.

docs/src/dev-docs/index.md

@@ -15,6 +15,13 @@ Building
 Docs about building CLP.
 :::
 
+:::{grid-item-card}
+:link: testing-presto-connector
+Presto connector
+^^^
+Building and testing the CLP Presto connector locally.
+:::
+
 :::{grid-item-card}
 :link: testing/index
 Testing
@@ -55,6 +62,7 @@ Any design docs describing parts of this project.
 :hidden:
 
 building-package
+testing-presto-connector
 :::
 
 :::{toctree}

docs/src/dev-docs/testing-presto-connector.md

@@ -0,0 +1,73 @@
+# Building and testing the Presto connector
+
+The Presto integration installs the [CLP connector][clp-connector] at startup from the
+`ghcr.io/y-scope/clp-plugin-presto-connector` image, so the stock Presto images stay unmodified.
+This page covers building that image locally and pointing the [Docker Compose](#docker-compose) and
+[Helm](#helm-kind) stacks at it.
+
+The default connector image is pinned by digest, so a local build of the same tag won't be picked
+up on its own — point `CLP_PRESTO_CONNECTOR_REF` at your image to use it.
+
+## Building the connector image
+
+In the [`clp-plugin-presto-connector`][clp-connector] repository, run:
+
+```shell
+task package
+```
+
+This builds `ghcr.io/y-scope/clp-plugin-presto-connector:0.1.0-SNAPSHOT` and loads it into your
+local Docker daemon.
+
+## Docker Compose
+
+The `presto-clp` stack is in `tools/deployment/presto-clp`. `scripts/set-up-config.sh` verifies the
+connector image exists (locally or on the registry) and writes it into `.env`, erroring with the
+ref it tried if neither exists.
+
+Each image is selected by a single environment variable holding a complete reference, which is used
+as-is. Any of `repository:tag`, `repository@digest`, or `repository:tag@digest` works; unset
+variables fall back to the pinned defaults.
+
+* `CLP_PRESTO_CONNECTOR_REF`: the CLP connector.
+* `CLP_PRESTO_COORDINATOR_REF`: the Presto coordinator.
+* `CLP_PRESTO_WORKER_REF`: the Presto worker.
+
+To run against the image you just built:
+
+```shell
+CLP_PRESTO_CONNECTOR_REF=ghcr.io/y-scope/clp-plugin-presto-connector:0.1.0-SNAPSHOT \
+    ./scripts/set-up-config.sh <clp-package-dir>
+docker compose up -d
+```
+
+Omitting the digest is what makes the local image win: Docker resolves a tag against the local
+daemon first, but a digest only ever matches the exact published image.
+
+See the [Using Presto with CLP][using-presto] user guide for the full setup.
+
+## Helm (kind)
+
+`--clp-connector-image` loads a local image into the `kind` cluster and sets
+`image.clpConnector.{repository,tag,pullPolicy=Never}` for you:
+
+```shell
+tools/deployment/package-helm/set-up-test.sh --presto \
+    --clp-connector-image ghcr.io/y-scope/clp-plugin-presto-connector:0.1.0-SNAPSHOT
+```
+
+## Pinning a new default
+
+When bumping a pinned default, take the digest of the *manifest list*, not of a per-architecture
+manifest — the connector is published for both `linux/amd64` and `linux/arm64`, and a per-arch
+digest would break the other architecture:
+
+```shell
+docker buildx imagetools inspect <ref> --format '{{.Manifest.Digest}}'
+```
+
+`docker inspect` reports the digest of the single-platform image you happen to have pulled, so it's
+the wrong source here.
+
+[clp-connector]: https://github.com/y-scope/clp-plugin-presto-connector
+[using-presto]: ../user-docs/guides-using-presto.md

docs/src/user-docs/guides-using-presto.md

@@ -102,6 +102,13 @@ When using Kubernetes, Presto worker scheduling can be configured using the
 Kubernetes deployment guide for details.
 :::
 
+:::{note}
+The CLP connector is installed at startup from the image under `image.clpConnector` (default
+`ghcr.io/y-scope/clp-plugin-presto-connector`). To test a locally-built connector image in a `kind`
+cluster, use the set-up script's `--clp-connector-image` flag; see
+[Testing the Presto connector locally][testing-presto-connector].
+:::
+
 ## Docker Compose
 
 ### Requirements
@@ -199,6 +206,15 @@ Using Presto with CLP via Docker Compose requires:
     * Replace `<clp-json-dir>` with the location of the clp-json package you set up in the previous
       section.
 
+    :::{note}
+    `set-up-config.sh` also verifies that the CLP connector image exists, locally or on the
+    registry, and writes it into `.env`. The default is pinned by digest, so the image can't change
+    underneath you. To use a different one, export `CLP_PRESTO_CONNECTOR_REF` before running the
+    script; it's used as-is, so it may be `repository:tag`, `repository@digest`, or
+    `repository:tag@digest`. See [Testing the Presto connector locally][testing-presto-connector]
+    for building the image yourself.
+    :::
+
 4. Configure Presto to use CLP's metadata database as follows:
 
     * Open and edit `coordinator/config-template/split-filter.json`.
@@ -332,3 +348,4 @@ These limitations will be addressed in a future release of the Presto integratio
 [Presto]: https://prestodb.io/
 [y-scope/presto#8]: https://github.com/y-scope/presto/issues/8
 [yscope-presto]: https://github.com/y-scope/presto
+[testing-presto-connector]: ../dev-docs/testing-presto-connector.md

tools/deployment/package-helm/.set-up-common.sh

@@ -58,7 +58,12 @@ get_image_helm_args() {
     fi
 
     echo "Loading local image '${image}' into kind cluster..." >&2
-    kind load docker-image "${image}" --name "${cluster_name}" >&2
+    # `errexit` is suspended inside the caller's `$(...) || exit 1`, so check explicitly;
+    # otherwise a failed load silently yields `pullPolicy=Never` for an unloaded image.
+    if ! kind load docker-image "${image}" --name "${cluster_name}" >&2; then
+        echo "Error: failed to load local image '${image}' into kind cluster '${cluster_name}'." >&2
+        return 1
+    fi
 
     # Split "repo:tag" on the last colon whose right-hand side contains no '/'
     # (so registry ports like localhost:5000/repo are not mistaken for tags).
@@ -75,11 +80,12 @@ get_image_helm_args() {
 }
 
 # Parses common arguments shared across set-up scripts.
-# Sets CLP_PACKAGE_IMAGE and ENABLE_PRESTO global variables.
+# Sets CLP_PACKAGE_IMAGE, CLP_PRESTO_CONNECTOR_IMAGE, and ENABLE_PRESTO global variables.
 #
 # @param {string[]} args Script arguments
 parse_common_args() {
     CLP_PACKAGE_IMAGE=""
+    CLP_PRESTO_CONNECTOR_IMAGE=""
     ENABLE_PRESTO="false"
     while [[ $# -gt 0 ]]; do
         case "$1" in
@@ -91,6 +97,14 @@ parse_common_args() {
                 CLP_PACKAGE_IMAGE="$2"
                 shift 2
                 ;;
+            --clp-connector-image)
+                if [[ $# -lt 2 || "$2" == --* ]]; then
+                    echo "Error: '--clp-connector-image' requires a value." >&2
+                    exit 1
+                fi
+                CLP_PRESTO_CONNECTOR_IMAGE="$2"
+                shift 2
+                ;;
             --presto)
                 ENABLE_PRESTO="true"
                 shift

tools/deployment/package-helm/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: "v2"
 name: "clp"
-version: "0.4.1-dev.3"
+version: "0.4.1-dev.4"
 description: "A Helm chart for CLP's (Compressed Log Processor) package deployment"
 type: "application"
 appVersion: "0.13.1-dev"

tools/deployment/package-helm/set-up-multi-dedicated-test.sh

@@ -136,8 +136,14 @@ EOF
 echo "Installing Helm chart..."
 helm uninstall test --ignore-not-found
 sleep 2
+
+# Resolve image overrides up front so an invalid or unloadable image exits loudly instead
+# of silently falling back to the chart default. An empty override resolves to no flags.
+clp_package_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}") || exit 1
+clp_connector_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpConnector" "${CLP_PRESTO_CONNECTOR_IMAGE}") || exit 1
+
 # Word splitting is intentional: helper functions return multiple --set flags.
-# shellcheck disable=SC2046
+# shellcheck disable=SC2086,SC2046
 helm install test "${script_dir}" \
     --set "distributedDeployment=true" \
     --set "scheduling.compressionWorker.replicas=${COMPRESSION_WORKER_REPLICAS}" \
@@ -162,6 +168,7 @@ helm install test "${script_dir}" \
     --set "scheduling.mcpServer.nodeSelector.yscope\.io/nodeType=core" \
     $(get_service_exposure_helm_args) \
     $(get_presto_helm_args) \
-    $(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}")
+    ${clp_package_args} \
+    ${clp_connector_args}
 
 wait_for_cluster_ready

tools/deployment/package-helm/set-up-multi-shared-test.sh

@@ -42,8 +42,14 @@ generate_kind_config "${NUM_WORKER_NODES}" | kind create cluster --name "${CLUST
 echo "Installing Helm chart..."
 helm uninstall test --ignore-not-found
 sleep 2
+
+# Resolve image overrides up front so an invalid or unloadable image exits loudly instead
+# of silently falling back to the chart default. An empty override resolves to no flags.
+clp_package_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}") || exit 1
+clp_connector_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpConnector" "${CLP_PRESTO_CONNECTOR_IMAGE}") || exit 1
+
 # Word splitting is intentional: helper functions return multiple --set flags.
-# shellcheck disable=SC2046
+# shellcheck disable=SC2086,SC2046
 helm install test "${script_dir}" \
     --set "distributedDeployment=true" \
     --set "scheduling.compressionWorker.replicas=${COMPRESSION_WORKER_REPLICAS}" \
@@ -52,6 +58,7 @@ helm install test "${script_dir}" \
     --set "scheduling.prestoWorker.replicas=${PRESTO_WORKER_REPLICAS}" \
     $(get_service_exposure_helm_args) \
     $(get_presto_helm_args) \
-    $(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}")
+    ${clp_package_args} \
+    ${clp_connector_args}
 
 wait_for_cluster_ready

tools/deployment/package-helm/set-up-test.sh

@@ -26,11 +26,18 @@ generate_kind_config 0 | kind create cluster --name "${CLUSTER_NAME}" --config=-
 echo "Installing Helm chart..."
 helm uninstall test --ignore-not-found
 sleep 2
+
+# Resolve image overrides up front so an invalid or unloadable image exits loudly instead
+# of silently falling back to the chart default. An empty override resolves to no flags.
+clp_package_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}") || exit 1
+clp_connector_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpConnector" "${CLP_PRESTO_CONNECTOR_IMAGE}") || exit 1
+
 # Word splitting is intentional: helper functions return multiple --set flags.
-# shellcheck disable=SC2046
+# shellcheck disable=SC2086,SC2046
 helm install test "${script_dir}" \
     $(get_service_exposure_helm_args) \
     $(get_presto_helm_args) \
-    $(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}")
+    ${clp_package_args} \
+    ${clp_connector_args}
 
 wait_for_cluster_ready

tools/deployment/package-helm/templates/_helpers.tpl

@@ -156,28 +156,32 @@ app.kubernetes.io/instance: {{ .Release.Name }}
 {{/*
 Creates a container image reference from .Values.image.
 
-Renders repository@digest when "digest" is set; otherwise, renders repository:tag. clpPackage
-defaults to Chart.AppVersion when "tag" is omitted; other components require "tag".
+Renders repository:tag, repository@digest, or repository:tag@digest, depending on which of "tag"
+and "digest" are set. Setting both keeps the tag as a human-readable label while the digest is what
+actually gets pulled. clpPackage defaults to Chart.AppVersion when "tag" is omitted; other
+components require at least one of "tag" or "digest".
 
 @param {object} root Root template context (required)
 @param {string} component Key under .Values.image (e.g., "clpPackage", "redis")
-@return {string} Full image reference (repository@digest or repository:tag)
+@return {string} Full image reference
 */}}
 {{- define "clp.imageRef" -}}
 {{- $img := index .root.Values.image .component -}}
-{{- if $img.digest -}}
-{{- printf "%s@%s" $img.repository $img.digest -}}
-{{- else -}}
 {{- $tag := $img.tag -}}
-{{- if not $tag -}}
-  {{- if eq .component "clpPackage" -}}
-    {{- $tag = .root.Chart.AppVersion -}}
-  {{- else -}}
-    {{- fail (printf "image.%s.tag is required" .component) -}}
-  {{- end -}}
+{{- if and (not $tag) (eq .component "clpPackage") -}}
+  {{- $tag = .root.Chart.AppVersion -}}
+{{- end -}}
+{{- if not (or $tag $img.digest) -}}
+  {{- fail (printf "image.%s requires \"tag\" or \"digest\"" .component) -}}
 {{- end -}}
-{{- printf "%s:%s" $img.repository $tag -}}
+{{- $ref := $img.repository -}}
+{{- if $tag -}}
+  {{- $ref = printf "%s:%s" $ref $tag -}}
+{{- end -}}
+{{- if $img.digest -}}
+  {{- $ref = printf "%s@%s" $ref $img.digest -}}
 {{- end -}}
+{{- $ref -}}
 {{- end }}
 
 {{/*

tools/deployment/package-helm/templates/configmap.yaml

@@ -391,6 +391,7 @@ data:
     inline-sql-functions=false
     nested-data-serialization-enabled=false
     native-execution-enabled=true
+    use-connector-provided-serialization-codecs=true
 
   presto-coordinator-config-jvm.config: |
     -server
@@ -438,6 +439,7 @@ data:
     system-memory-gb={{ .worker.system_memory_gb }}
     register-test-functions=false
     runtime-metrics-collection-enabled=false
+    plugin.dir=/opt/presto-server/plugin/clp
 
   presto-worker-config-node.properties: |
     node.environment=production

tools/deployment/package-helm/templates/presto-coordinator-deployment.yaml

@@ -30,9 +30,22 @@ spec:
             "type" "job"
             "name" "db-table-creator"
           ) | nindent 10 }}
+        - name: "install-clp-plugin"
+          image: {{ include "clp.imageRef" (dict "root" . "component" "clpConnector") | quote }}
+          imagePullPolicy: {{ .Values.image.clpConnector.pullPolicy | quote }}
+          env:
+            - name: "COORDINATOR_PLUGIN_INSTALL_PATH"
+              value: "/install/coordinator"
+          {{- include "clp.createResourceLimits" (dict
+              "root" .
+              "component" "prestoCoordinatorInstallPlugin"
+            ) | nindent 10 }}
+          volumeMounts:
+            - name: "presto-plugin"
+              mountPath: "/install/coordinator"
       containers:
         - name: "presto-coordinator"
-          image: "{{ .Values.image.prestoCoordinator.repository }}:{{ .Values.image.prestoCoordinator.tag }}"
+          image: {{ include "clp.imageRef" (dict "root" . "component" "prestoCoordinator") | quote }}
           imagePullPolicy: "{{ .Values.image.prestoCoordinator.pullPolicy }}"
           {{- include "clp.createResourceLimits" (dict
               "root" .
@@ -65,6 +78,9 @@ spec:
               mountPath: "/opt/presto-server/etc/split-filter.json"
               subPath: "presto-coordinator-config-split-filter.json"
               readOnly: true
+            - name: "presto-plugin"
+              mountPath: "/opt/presto-server/plugin/clp"
+              readOnly: true
           readinessProbe:
             {{- include "clp.readinessProbeTimings" . | nindent 12 }}
             httpGet: &presto-coordinator-health-check
@@ -83,4 +99,6 @@ spec:
         - name: "presto-config"
           configMap:
             name: {{ include "clp.fullname" . }}-config
+        - name: "presto-plugin"
+          emptyDir: {}
 {{- end }}