Skip to content

feat(packaging): Add a busybox init-container installer image as a fourth distribution channel. - #36

Merged
jackluo923 merged 10 commits into
feat/ci-cpu-target-inputsfrom
feat/init-container-installer
Aug 2, 2026
Merged

feat(packaging): Add a busybox init-container installer image as a fourth distribution channel.#36
jackluo923 merged 10 commits into
feat/ci-cpu-target-inputsfrom
feat/init-container-installer

Conversation

@jackluo923

@jackluo923 jackluo923 commented Jul 28, 2026

Copy link
Copy Markdown
Member

Description

Adds a fourth distribution channel alongside the .deb/.rpm/.tar.gz packages: a busybox init-container installer image that bundles both plugins and copies each into a mounted volume at pod init, so the stock presto-native images can be used unmodified (how the connector is consumed in CLP-OSS and CLP cloud deployments; y-scope/clp#2439 is the deployment counterpart).

Each plugin is selected by its own env var, since the coordinator JAR and worker .so install into different locations (set either or both):

docker run --rm -e WORKER_PLUGIN_INSTALL_PATH=/plugins -v "$(pwd)/plugins:/plugins" \
  ghcr.io/y-scope/clp-plugin-presto-connector:<version>

CI builds the image per architecture on every run and assembles a multi-arch :<version> tag, but pushes to GHCR only from the default branch and version tags. Local builds (task package) load the building machine's :<version>-<arch> tag. Details in tools/build-packages/README.md.

Validation performed

  • Built via build-installer-init-image.sh --load from a locally built package tarball; a run with both COORDINATOR_PLUGIN_INSTALL_PATH and WORKER_PLUGIN_INSTALL_PATH set installed the JAR and the .so + lib/ into the right targets (file list matches the tarball exactly); a run with neither set exited with the usage error.
  • Non-root run (--user "$(id -u):$(id -g)"): installs cleanly, files owned by the running user.

@jackluo923
jackluo923 requested review from a team and 20001020ycx as code owners July 28, 2026 05:51
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2e99db94-4490-4ed7-a92e-851dcc4c46a3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/init-container-installer

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jackluo923 jackluo923 changed the title feat(packaging): Add a busybox init-container installer image and dependency-pin validation. feat(packaging): Add a busybox init-container installer image as a fourth distribution channel. Jul 28, 2026
@jackluo923
jackluo923 force-pushed the feat/init-container-installer branch 2 times, most recently from 2faa4fa to 06d8229 Compare July 30, 2026 21:13
@jackluo923
jackluo923 changed the base branch from feat/derive-worker-cxx-flags to feat/ci-cpu-target-inputs July 30, 2026 21:13
@jackluo923
jackluo923 force-pushed the feat/init-container-installer branch from 06d8229 to 8f591e1 Compare July 30, 2026 21:14
@jackluo923
jackluo923 requested a review from junhaoliao July 31, 2026 01:26
jackluo923 and others added 3 commits July 31, 2026 15:12
…urth distribution channel.

Builds a small busybox image bundling both plugins under `/opt/clp-plugin-presto-connector`; its entrypoint copies the coordinator JAR and/or native worker `.so` into mounted volumes named by `COORDINATOR_PLUGIN_INSTALL_PATH` / `WORKER_PLUGIN_INSTALL_PATH` (at least one required, else it errors), so a coordinator pod and a worker pod run the same image with different config.

`build-installer-init-image.sh` builds the image host-side from a package tarball (docker isn't available inside the build-env container). `task package` now also builds and loads it locally, and the CI workflow builds it per architecture on native runners and publishes a multi-arch `:<version>` manifest from the default branch and tags.

Moves `image_repo_from_origin` into `dependency-image/utils.sh` so both `build-dependency-image.sh` and the new script share one GHCR-repo derivation.
@jackluo923
jackluo923 force-pushed the feat/init-container-installer branch from 93a61cb to 65737b0 Compare July 31, 2026 19:12

@junhaoliao junhaoliao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as debugged offline, there's one permission issue we need to fix

the rest lgtm

Comment thread tools/build-packages/build-installer-init-image.sh
image_repo="ghcr.io/$(printf '%s' "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]')"
# Docker tags allow only [A-Za-z0-9_.-]; sanitize to match the build script.
tag_version="${VERSION//[^A-Za-z0-9_.-]/_}"
docker buildx imagetools create \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we consider this -

the architecture tags are mutable shared names, so overlapping runs can interleave before this manifest resolves them. publish run-scoped staging tags, then promote only those references after both matrix legs succeed:

# image job
staging_version="${VERSION}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
# pass --version "${staging_version}" and --push to the builder

# manifest job
source_tag="${tag_version}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
docker buildx imagetools create --prefer-index=false \
  --tag "${image_repo}:${tag_version}-amd64" "${image_repo}:${source_tag}-amd64"
docker buildx imagetools create --prefer-index=false \
  --tag "${image_repo}:${tag_version}-arm64" "${image_repo}:${source_tag}-arm64"
docker buildx imagetools create --tag "${image_repo}:${tag_version}" \
  "${image_repo}:${source_tag}-amd64" "${image_repo}:${source_tag}-arm64"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — the race is real: the per-arch tags are mutable shared names, so two overlapping publish runs (e.g. a re-run of an older attempt, or a tag build racing main) could produce a manifest mixing images from different runs.

Rather than run-scoped staging tags, I went with referencing the images by digest: each image leg captures the pushed image's registry digest (buildx --metadata-file) and exposes it as a job output, and the manifest job does imagetools create --tag : @ @. Same guarantee — the manifest combines exactly the images this run built — but:

  • digests are immutable, so there's no window at all (staging tags are still mutable names, just less likely to collide);
  • no staging tags accumulating in the GHCR package (they'd need a cleanup step to avoid piling up per run);
  • no extra pushes — the digest is a side effect of the push we already do.

The per-arch :- tags are still pushed for debugging convenience; they just no longer carry correctness. The final : tag stays mutable by design — that's the SNAPSHOT contract (each publish is supposed to move it); the fix only ensures each move is internally consistent.

[[ -n "${repo}" ]] || repo="$(image_repo_from_origin)"

# Docker tags allow only [A-Za-z0-9_.-]; sanitize any other version characters (e.g. '+').
tag_version="${version//[^A-Za-z0-9_.-]/_}"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this replacement is lossy: valid versions 1.0+rc and 1.0~rc both become 1.0_rc. define one shared encoding and use it here and in the manifest job:

package_version_to_image_tag() {
    local version="$1"
    [[ "${version}" =~ ^[0-9][0-9A-Za-z.+~-]*$ ]] || return 1
    local tag_version="${version//+/_plus_}"
    tag_version="${tag_version//\~/_tilde_}"
    printf '%s\n' "${tag_version}"
}

tag_version="$(package_version_to_image_tag "${version}")" ||
    die "invalid package version: ${version}"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Connector version shouldn't contain "+" or "~". Mistake on my part. I will switch to fail loudly if these versions are encountered.

Comment thread .github/workflows/build-packages.yaml Outdated
Comment thread tools/build-packages/README.md Outdated
Comment thread tools/build-packages/image/Dockerfile
Comment thread tools/build-packages/build-installer-init-image.sh Outdated
jackluo923 and others added 6 commits July 31, 2026 19:57
Co-authored-by: Junhao Liao <junhao@junhao.ca>
Co-authored-by: Junhao Liao <junhao@junhao.ca>
Co-authored-by: Junhao Liao <junhao@junhao.ca>
…t-image.sh.

Co-authored-by: Junhao Liao <junhao@junhao.ca>
…of mangling them

The old inline sanitize (${VERSION//[^A-Za-z0-9_.-]/_}) was lossy: distinct
package versions like '1.0+rc' and '1.0~rc' both mapped to the same '1.0_rc'
tag. Replace it with a shared package_version_to_image_tag helper in
dependency-image/utils.sh that validates the version and fails loudly when it
contains characters Docker tags can't represent ('+', '~'). Both the local
build script and the CI manifest job now use the same helper, so the manifest
tag always matches the per-arch tags.

Co-authored-by: Junhao Liao <junhao@junhao.ca>
…rch> tag suffix

Manifest by digest: the manifest job previously combined the per-arch images by
their mutable :<version>-<arch> tags, so a concurrent publish run could move a
tag between the image jobs and the manifest job, silently mixing images from
different runs. Each image leg now captures the pushed image's registry digest
(build-installer-init-image.sh --digest-file, via buildx --metadata-file) and
exposes it as a job output; the manifest job references the per-arch images by
those immutable digests, guaranteeing the manifest combines exactly the images
this run built. Per-arch tags are still pushed for debugging convenience but no
longer carry correctness.

Local tag without arch suffix: --load now tags the image :<version> instead of
:<version>-<arch>, the conventional Docker pattern where a locally-built image
and the published multi-arch one share a name and whatever is in the daemon
wins. The suffix remains only on pushed images, where the two CI legs need
distinct registry names below the manifest.

Co-authored-by: Junhao Liao <junhao@junhao.ca>
@jackluo923
jackluo923 requested a review from junhaoliao August 1, 2026 00:40

@junhaoliao junhaoliao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made one modification to the docs directly

@jackluo923
jackluo923 merged commit ec70485 into main Aug 2, 2026
9 checks passed
@jackluo923
jackluo923 deleted the feat/init-container-installer branch August 2, 2026 04:27
jackluo923 added a commit that referenced this pull request Aug 2, 2026
… docs.

Rebasing onto main brought in the init-container installer image (#36), which
these docs predate:

- `tools/README.md` described `build-packages/` as producing three distribution
  channels; it now produces four.
- The Layout section omitted `build-installer-init-image.sh` and `image/`.
- `dependency-image/utils.sh` has since absorbed `image_repo_from_origin` from
  `build-dependency-image.sh` and gained `package_version_to_image_tag`, so it
  is now shared with `build-installer-init-image.sh` too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants