feat(deployment): Deploy Presto with stock images and install the CLP connector as a plugin.

docs/src/dev-docs/index.md

@@ -15,6 +15,13 @@ Building
 Docs about building CLP.
 :::
 
+:::{grid-item-card}
+:link: testing-presto-connector
+Presto connector
+^^^
+Building and testing the CLP Presto connector locally.
+:::
+
 :::{grid-item-card}
 :link: testing/index
 Testing
@@ -55,6 +62,7 @@ Any design docs describing parts of this project.
 :hidden:
 
 building-package
+testing-presto-connector
 :::
 
 :::{toctree}

docs/src/dev-docs/testing-presto-connector.md

@@ -0,0 +1,52 @@
+# Building and testing the Presto connector
+
+The Presto integration installs the [CLP connector][clp-connector] at startup from the
+`ghcr.io/y-scope/clp-plugin-presto-connector` image, so the stock Presto images stay unmodified.
+This page covers building that image locally and pointing the [Docker Compose](#docker-compose) and
+[Helm](#helm-kind) stacks at it.
+
+Local and published images share the same `:<version>` tag (e.g. `0.1.0-SNAPSHOT`), so whatever is
+in your Docker daemon wins. `docker rmi` your local build to go back to the published image.
+
+## Building the connector image
+
+In the [`clp-plugin-presto-connector`][clp-connector] repository, run:
+
+```shell
+task package
+```
+
+This builds `ghcr.io/y-scope/clp-plugin-presto-connector:0.1.0-SNAPSHOT` and loads it into your
+local Docker daemon.
+
+## Docker Compose
+
+The `presto-clp` stack is in `tools/deployment/presto-clp`. `scripts/set-up-config.sh` verifies the
+connector image exists (locally or on the registry) and writes it into `.env`, erroring with the
+ref it tried if neither exists. Export any of these before running it:
+
+* `CLP_PRESTO_CONNECTOR_IMAGE`: repository (default `ghcr.io/y-scope/clp-plugin-presto-connector`).
+* `CLP_PRESTO_CONNECTOR_VERSION`: tag to verify and use (default `0.1.0-SNAPSHOT`).
+* `CLP_PRESTO_CONNECTOR_TAG`: exact tag, skipping the existence check.
+
+Then start the stack:
+
+```shell
+./scripts/set-up-config.sh <clp-package-dir>
+docker compose up -d
+```
+
+See the [Using Presto with CLP][using-presto] user guide for the full setup.
+
+## Helm (kind)
+
+`--clp-connector-image` loads a local image into the `kind` cluster and sets
+`image.clpConnector.{repository,tag,pullPolicy=Never}` for you:
+
+```shell
+tools/deployment/package-helm/set-up-test.sh --presto \
+    --clp-connector-image ghcr.io/y-scope/clp-plugin-presto-connector:0.1.0-SNAPSHOT
+```
+
+[clp-connector]: https://github.com/y-scope/clp-plugin-presto-connector
+[using-presto]: ../user-docs/guides-using-presto.md

docs/src/user-docs/guides-using-presto.md

@@ -102,6 +102,13 @@ When using Kubernetes, Presto worker scheduling can be configured using the
 Kubernetes deployment guide for details.
 :::
 
+:::{note}
+The CLP connector is installed at startup from the image under `image.clpConnector` (default
+`ghcr.io/y-scope/clp-plugin-presto-connector`). To test a locally-built connector image in a `kind`
+cluster, use the set-up script's `--clp-connector-image` flag; see
+[Testing the Presto connector locally][testing-presto-connector].
+:::
+
 ## Docker Compose
 
 ### Requirements
@@ -199,6 +206,15 @@ Using Presto with CLP via Docker Compose requires:
     * Replace `<clp-json-dir>` with the location of the clp-json package you set up in the previous
       section.
 
+    :::{note}
+    `set-up-config.sh` also verifies that the CLP connector image exists, locally or on the
+    registry, and writes it into `.env`. To override the image, export `CLP_PRESTO_CONNECTOR_IMAGE`
+    or `CLP_PRESTO_CONNECTOR_VERSION` before running the script. `CLP_PRESTO_CONNECTOR_TAG` sets an
+    exact tag and skips the check, so an invalid tag surfaces later during `docker compose up`. See
+    [Testing the Presto connector locally][testing-presto-connector] for building the image
+    yourself.
+    :::
+
 4. Configure Presto to use CLP's metadata database as follows:
 
     * Open and edit `coordinator/config-template/split-filter.json`.
@@ -332,3 +348,4 @@ These limitations will be addressed in a future release of the Presto integratio
 [Presto]: https://prestodb.io/
 [y-scope/presto#8]: https://github.com/y-scope/presto/issues/8
 [yscope-presto]: https://github.com/y-scope/presto
+[testing-presto-connector]: ../dev-docs/testing-presto-connector.md

tools/deployment/package-helm/.set-up-common.sh

@@ -58,7 +58,15 @@ get_image_helm_args() {
     fi
 
     echo "Loading local image '${image}' into kind cluster..." >&2
-    kind load docker-image "${image}" --name "${cluster_name}" >&2
+    # Check explicitly rather than relying on `errexit`: callers capture this
+    # function's output via `$(...) ... || exit 1`, and the `||` suspends errexit
+    # inside the command substitution — so an unchecked `kind load` failure would
+    # be silently ignored and we'd proceed with `pullPolicy=Never` for an image
+    # that was never loaded.
+    if ! kind load docker-image "${image}" --name "${cluster_name}" >&2; then
+        echo "Error: failed to load local image '${image}' into kind cluster '${cluster_name}'." >&2
+        return 1
+    fi
 
     # Split "repo:tag" on the last colon whose right-hand side contains no '/'
     # (so registry ports like localhost:5000/repo are not mistaken for tags).
@@ -75,11 +83,12 @@ get_image_helm_args() {
 }
 
 # Parses common arguments shared across set-up scripts.
-# Sets CLP_PACKAGE_IMAGE and ENABLE_PRESTO global variables.
+# Sets CLP_PACKAGE_IMAGE, CLP_PRESTO_CONNECTOR_IMAGE, and ENABLE_PRESTO global variables.
 #
 # @param {string[]} args Script arguments
 parse_common_args() {
     CLP_PACKAGE_IMAGE=""
+    CLP_PRESTO_CONNECTOR_IMAGE=""
     ENABLE_PRESTO="false"
     while [[ $# -gt 0 ]]; do
         case "$1" in
@@ -91,6 +100,14 @@ parse_common_args() {
                 CLP_PACKAGE_IMAGE="$2"
                 shift 2
                 ;;
+            --clp-connector-image)
+                if [[ $# -lt 2 || "$2" == --* ]]; then
+                    echo "Error: '--clp-connector-image' requires a value." >&2
+                    exit 1
+                fi
+                CLP_PRESTO_CONNECTOR_IMAGE="$2"
+                shift 2
+                ;;
             --presto)
                 ENABLE_PRESTO="true"
                 shift

tools/deployment/package-helm/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: "v2"
 name: "clp"
-version: "0.4.1-dev.3"
+version: "0.4.1-dev.4"
 description: "A Helm chart for CLP's (Compressed Log Processor) package deployment"
 type: "application"
 appVersion: "0.13.1-dev"

tools/deployment/package-helm/set-up-multi-dedicated-test.sh

@@ -136,8 +136,17 @@ EOF
 echo "Installing Helm chart..."
 helm uninstall test --ignore-not-found
 sleep 2
+
+# Resolve the local-image overrides into Helm --set flags up front so a failure
+# (an invalid image ref, or an image absent from the local Docker daemon) exits
+# loudly instead of being silently dropped — which would make `helm install`
+# fall back to the chart-default image. An empty override is intentional (no
+# --clp-*-image passed) and resolves to empty flags.
+clp_package_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}") || exit 1
+clp_connector_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpConnector" "${CLP_PRESTO_CONNECTOR_IMAGE}") || exit 1
+
 # Word splitting is intentional: helper functions return multiple --set flags.
-# shellcheck disable=SC2046
+# shellcheck disable=SC2086,SC2046
 helm install test "${script_dir}" \
     --set "distributedDeployment=true" \
     --set "scheduling.compressionWorker.replicas=${COMPRESSION_WORKER_REPLICAS}" \
@@ -162,6 +171,7 @@ helm install test "${script_dir}" \
     --set "scheduling.mcpServer.nodeSelector.yscope\.io/nodeType=core" \
     $(get_service_exposure_helm_args) \
     $(get_presto_helm_args) \
-    $(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}")
+    ${clp_package_args} \
+    ${clp_connector_args}
 
 wait_for_cluster_ready

tools/deployment/package-helm/set-up-multi-shared-test.sh

@@ -42,8 +42,16 @@ generate_kind_config "${NUM_WORKER_NODES}" | kind create cluster --name "${CLUST
 echo "Installing Helm chart..."
 helm uninstall test --ignore-not-found
 sleep 2
+# Resolve the local-image overrides into Helm --set flags up front so a failure
+# (an invalid image ref, or an image absent from the local Docker daemon) exits
+# loudly instead of being silently dropped — which would make `helm install`
+# fall back to the chart-default image. An empty override is intentional (no
+# --clp-*-image passed) and resolves to empty flags.
+clp_package_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}") || exit 1
+clp_connector_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpConnector" "${CLP_PRESTO_CONNECTOR_IMAGE}") || exit 1
+
 # Word splitting is intentional: helper functions return multiple --set flags.
-# shellcheck disable=SC2046
+# shellcheck disable=SC2086,SC2046
 helm install test "${script_dir}" \
     --set "distributedDeployment=true" \
     --set "scheduling.compressionWorker.replicas=${COMPRESSION_WORKER_REPLICAS}" \
@@ -52,6 +60,7 @@ helm install test "${script_dir}" \
     --set "scheduling.prestoWorker.replicas=${PRESTO_WORKER_REPLICAS}" \
     $(get_service_exposure_helm_args) \
     $(get_presto_helm_args) \
-    $(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}")
+    ${clp_package_args} \
+    ${clp_connector_args}
 
 wait_for_cluster_ready

tools/deployment/package-helm/set-up-test.sh

@@ -26,11 +26,21 @@ generate_kind_config 0 | kind create cluster --name "${CLUSTER_NAME}" --config=-
 echo "Installing Helm chart..."
 helm uninstall test --ignore-not-found
 sleep 2
+
+# Resolve the local-image overrides into Helm --set flags up front so a failure
+# (an invalid image ref, or an image absent from the local Docker daemon) exits
+# loudly instead of being silently dropped — which would make `helm install`
+# fall back to the chart-default image. An empty override is intentional (no
+# --clp-*-image passed) and resolves to empty flags.
+clp_package_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}") || exit 1
+clp_connector_args=$(get_image_helm_args "${CLUSTER_NAME}" "clpConnector" "${CLP_PRESTO_CONNECTOR_IMAGE}") || exit 1
+
 # Word splitting is intentional: helper functions return multiple --set flags.
-# shellcheck disable=SC2046
+# shellcheck disable=SC2086,SC2046
 helm install test "${script_dir}" \
     $(get_service_exposure_helm_args) \
     $(get_presto_helm_args) \
-    $(get_image_helm_args "${CLUSTER_NAME}" "clpPackage" "${CLP_PACKAGE_IMAGE}")
+    ${clp_package_args} \
+    ${clp_connector_args}
 
 wait_for_cluster_ready

tools/deployment/package-helm/templates/configmap.yaml

@@ -391,6 +391,7 @@ data:
     inline-sql-functions=false
     nested-data-serialization-enabled=false
     native-execution-enabled=true
+    use-connector-provided-serialization-codecs=true
 
   presto-coordinator-config-jvm.config: |
     -server
@@ -438,6 +439,7 @@ data:
     system-memory-gb={{ .worker.system_memory_gb }}
     register-test-functions=false
     runtime-metrics-collection-enabled=false
+    plugin.dir=/opt/presto-server/plugin/clp
 
   presto-worker-config-node.properties: |
     node.environment=production

tools/deployment/package-helm/templates/presto-coordinator-deployment.yaml

@@ -30,6 +30,19 @@ spec:
             "type" "job"
             "name" "db-table-creator"
           ) | nindent 10 }}
+        - name: "install-clp-plugin"
+          image: {{ include "clp.imageRef" (dict "root" . "component" "clpConnector") | quote }}
+          imagePullPolicy: {{ .Values.image.clpConnector.pullPolicy | quote }}
+          env:
+            - name: "COORDINATOR_PLUGIN_INSTALL_PATH"
+              value: "/install/coordinator"
+          {{- include "clp.createResourceLimits" (dict
+              "root" .
+              "component" "prestoCoordinatorInstallPlugin"
+            ) | nindent 10 }}
+          volumeMounts:
+            - name: "presto-plugin"
+              mountPath: "/install/coordinator"
       containers:
         - name: "presto-coordinator"
           image: "{{ .Values.image.prestoCoordinator.repository }}:{{ .Values.image.prestoCoordinator.tag }}"
@@ -65,6 +78,9 @@ spec:
               mountPath: "/opt/presto-server/etc/split-filter.json"
               subPath: "presto-coordinator-config-split-filter.json"
               readOnly: true
+            - name: "presto-plugin"
+              mountPath: "/opt/presto-server/plugin/clp"
+              readOnly: true
           readinessProbe:
             {{- include "clp.readinessProbeTimings" . | nindent 12 }}
             httpGet: &presto-coordinator-health-check
@@ -83,4 +99,6 @@ spec:
         - name: "presto-config"
           configMap:
             name: {{ include "clp.fullname" . }}-config
+        - name: "presto-plugin"
+          emptyDir: {}
 {{- end }}

tools/deployment/package-helm/templates/presto-worker-deployment.yaml

@@ -30,6 +30,19 @@ spec:
             "type" "service"
             "name" "presto-coordinator"
           ) | nindent 10 }}
+        - name: "install-clp-plugin"
+          image: {{ include "clp.imageRef" (dict "root" . "component" "clpConnector") | quote }}
+          imagePullPolicy: {{ .Values.image.clpConnector.pullPolicy | quote }}
+          env:
+            - name: "WORKER_PLUGIN_INSTALL_PATH"
+              value: "/install/worker"
+          {{- include "clp.createResourceLimits" (dict
+              "root" .
+              "component" "prestoWorkerInstallPlugin"
+            ) | nindent 10 }}
+          volumeMounts:
+            - name: "presto-plugin"
+              mountPath: "/install/worker"
         - name: "setup-configs"
           image: {{ include "clp.imageRef" (dict "root" . "component" "kubectl") | quote }}
           imagePullPolicy: {{ .Values.image.kubectl.pullPolicy | quote }}
@@ -68,6 +81,9 @@ spec:
               readOnly: true
             - name: "presto-etc"
               mountPath: "/opt/presto-server/etc"
+            - name: "presto-plugin"
+              mountPath: "/opt/presto-server/plugin/clp"
+              readOnly: true
             {{- if eq .Values.clpConfig.archive_output.storage.type "fs" }}
             - name: {{ include "clp.volumeName" (dict
                 "component_category" "shared-data"
@@ -95,6 +111,8 @@ spec:
             name: {{ include "clp.fullname" . }}-config
         - name: "presto-etc"
           emptyDir: {}
+        - name: "presto-plugin"
+          emptyDir: {}
         - name: "presto-scripts"
           configMap:
             name: {{ include "clp.fullname" . }}-config

tools/deployment/package-helm/values.yaml

@@ -5,6 +5,12 @@ fullnameOverride: ""
 allowHostAccessForSbinScripts: true
 
 image:
+  # Installs the CLP Presto connector plugin into the Presto coordinator and worker pods. Override
+  # the repository to use a different edition of the connector.
+  clpConnector:
+    repository: "ghcr.io/y-scope/clp-plugin-presto-connector"
+    pullPolicy: "IfNotPresent"
+    tag: "0.1.0-SNAPSHOT"
   clpPackage:
     repository: "ghcr.io/y-scope/clp/clp-package"
     pullPolicy: "Always"
@@ -31,13 +37,13 @@ image:
     pullPolicy: "IfNotPresent"
     tag: "0.152.0"
   prestoCoordinator:
-    repository: "ghcr.io/y-scope/presto/coordinator"
+    repository: "ghcr.io/y-scope/presto"
     pullPolicy: "IfNotPresent"
-    tag: "clp-v0.10.0"
+    tag: "0.299"
   prestoWorker:
-    repository: "ghcr.io/y-scope/presto/prestissimo-worker"
+    repository: "ghcr.io/y-scope/presto-native"
     pullPolicy: "IfNotPresent"
-    tag: "clp-v0.10.0-fix.1"
+    tag: "0.299"
   queue:
     repository: "rabbitmq"
     pullPolicy: "Always"
@@ -142,6 +148,8 @@ resources:
 
   # Jobs
   dbTableCreator: {}
+  prestoCoordinatorInstallPlugin: {}
+  prestoWorkerInstallPlugin: {}
   prestoWorkerSetupConfigs: {}
   resultsCacheIndicesCreator: {}
   topologyMetricsEmitter: {}

tools/deployment/presto-clp/coordinator/config-template/config.properties

@@ -11,3 +11,4 @@ use-alternative-function-signatures=true
 inline-sql-functions=false
 nested-data-serialization-enabled=false
 native-execution-enabled=true
+use-connector-provided-serialization-codecs=true

tools/deployment/presto-clp/coordinator/scripts/generate-configs.sh

@@ -15,6 +15,6 @@ find /configs -type f | while read -r f; do
     ) | sh >"${PRESTO_CONFIG_DIR}/$(basename "$f")"
 done
 
-# Remove existing catalog files that exist in the image and add the CLP catalog
-rm -f "${PRESTO_CONFIG_DIR}/catalog/"*
+# Create the catalog directory and add the CLP catalog
+mkdir -p "${PRESTO_CONFIG_DIR}/catalog"
 mv "${PRESTO_CONFIG_DIR}/clp.properties" "${PRESTO_CONFIG_DIR}/catalog"