feat(packaging): Add a busybox init-container installer image as a fourth distribution channel. - #36
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2faa4fa to
06d8229
Compare
06d8229 to
8f591e1
Compare
…urth distribution channel. Builds a small busybox image bundling both plugins under `/opt/clp-plugin-presto-connector`; its entrypoint copies the coordinator JAR and/or native worker `.so` into mounted volumes named by `COORDINATOR_PLUGIN_INSTALL_PATH` / `WORKER_PLUGIN_INSTALL_PATH` (at least one required, else it errors), so a coordinator pod and a worker pod run the same image with different config. `build-installer-init-image.sh` builds the image host-side from a package tarball (docker isn't available inside the build-env container). `task package` now also builds and loads it locally, and the CI workflow builds it per architecture on native runners and publishes a multi-arch `:<version>` manifest from the default branch and tags. Moves `image_repo_from_origin` into `dependency-image/utils.sh` so both `build-dependency-image.sh` and the new script share one GHCR-repo derivation.
…ns, so a non-root init container works.
93a61cb to
65737b0
Compare
junhaoliao
left a comment
There was a problem hiding this comment.
as debugged offline, there's one permission issue we need to fix
the rest lgtm
| image_repo="ghcr.io/$(printf '%s' "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]')" | ||
| # Docker tags allow only [A-Za-z0-9_.-]; sanitize to match the build script. | ||
| tag_version="${VERSION//[^A-Za-z0-9_.-]/_}" | ||
| docker buildx imagetools create \ |
There was a problem hiding this comment.
shall we consider this -
the architecture tags are mutable shared names, so overlapping runs can interleave before this manifest resolves them. publish run-scoped staging tags, then promote only those references after both matrix legs succeed:
# image job
staging_version="${VERSION}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
# pass --version "${staging_version}" and --push to the builder
# manifest job
source_tag="${tag_version}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
docker buildx imagetools create --prefer-index=false \
--tag "${image_repo}:${tag_version}-amd64" "${image_repo}:${source_tag}-amd64"
docker buildx imagetools create --prefer-index=false \
--tag "${image_repo}:${tag_version}-arm64" "${image_repo}:${source_tag}-arm64"
docker buildx imagetools create --tag "${image_repo}:${tag_version}" \
"${image_repo}:${source_tag}-amd64" "${image_repo}:${source_tag}-arm64"There was a problem hiding this comment.
Good catch — the race is real: the per-arch tags are mutable shared names, so two overlapping publish runs (e.g. a re-run of an older attempt, or a tag build racing main) could produce a manifest mixing images from different runs.
Rather than run-scoped staging tags, I went with referencing the images by digest: each image leg captures the pushed image's registry digest (buildx --metadata-file) and exposes it as a job output, and the manifest job does imagetools create --tag : @ @. Same guarantee — the manifest combines exactly the images this run built — but:
- digests are immutable, so there's no window at all (staging tags are still mutable names, just less likely to collide);
- no staging tags accumulating in the GHCR package (they'd need a cleanup step to avoid piling up per run);
- no extra pushes — the digest is a side effect of the push we already do.
The per-arch :- tags are still pushed for debugging convenience; they just no longer carry correctness. The final : tag stays mutable by design — that's the SNAPSHOT contract (each publish is supposed to move it); the fix only ensures each move is internally consistent.
| [[ -n "${repo}" ]] || repo="$(image_repo_from_origin)" | ||
|
|
||
| # Docker tags allow only [A-Za-z0-9_.-]; sanitize any other version characters (e.g. '+'). | ||
| tag_version="${version//[^A-Za-z0-9_.-]/_}" |
There was a problem hiding this comment.
this replacement is lossy: valid versions 1.0+rc and 1.0~rc both become 1.0_rc. define one shared encoding and use it here and in the manifest job:
package_version_to_image_tag() {
local version="$1"
[[ "${version}" =~ ^[0-9][0-9A-Za-z.+~-]*$ ]] || return 1
local tag_version="${version//+/_plus_}"
tag_version="${tag_version//\~/_tilde_}"
printf '%s\n' "${tag_version}"
}
tag_version="$(package_version_to_image_tag "${version}")" ||
die "invalid package version: ${version}"There was a problem hiding this comment.
Connector version shouldn't contain "+" or "~". Mistake on my part. I will switch to fail loudly if these versions are encountered.
Co-authored-by: Junhao Liao <junhao@junhao.ca>
Co-authored-by: Junhao Liao <junhao@junhao.ca>
Co-authored-by: Junhao Liao <junhao@junhao.ca>
…t-image.sh. Co-authored-by: Junhao Liao <junhao@junhao.ca>
…of mangling them
The old inline sanitize (${VERSION//[^A-Za-z0-9_.-]/_}) was lossy: distinct
package versions like '1.0+rc' and '1.0~rc' both mapped to the same '1.0_rc'
tag. Replace it with a shared package_version_to_image_tag helper in
dependency-image/utils.sh that validates the version and fails loudly when it
contains characters Docker tags can't represent ('+', '~'). Both the local
build script and the CI manifest job now use the same helper, so the manifest
tag always matches the per-arch tags.
Co-authored-by: Junhao Liao <junhao@junhao.ca>
…rch> tag suffix Manifest by digest: the manifest job previously combined the per-arch images by their mutable :<version>-<arch> tags, so a concurrent publish run could move a tag between the image jobs and the manifest job, silently mixing images from different runs. Each image leg now captures the pushed image's registry digest (build-installer-init-image.sh --digest-file, via buildx --metadata-file) and exposes it as a job output; the manifest job references the per-arch images by those immutable digests, guaranteeing the manifest combines exactly the images this run built. Per-arch tags are still pushed for debugging convenience but no longer carry correctness. Local tag without arch suffix: --load now tags the image :<version> instead of :<version>-<arch>, the conventional Docker pattern where a locally-built image and the published multi-arch one share a name and whatever is in the daemon wins. The suffix remains only on pushed images, where the two CI legs need distinct registry names below the manifest. Co-authored-by: Junhao Liao <junhao@junhao.ca>
junhaoliao
left a comment
There was a problem hiding this comment.
made one modification to the docs directly
… docs. Rebasing onto main brought in the init-container installer image (#36), which these docs predate: - `tools/README.md` described `build-packages/` as producing three distribution channels; it now produces four. - The Layout section omitted `build-installer-init-image.sh` and `image/`. - `dependency-image/utils.sh` has since absorbed `image_repo_from_origin` from `build-dependency-image.sh` and gained `package_version_to_image_tag`, so it is now shared with `build-installer-init-image.sh` too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Description
Adds a fourth distribution channel alongside the
.deb/.rpm/.tar.gzpackages: a busybox init-container installer image that bundles both plugins and copies each into a mounted volume at pod init, so the stock presto-native images can be used unmodified (how the connector is consumed in CLP-OSS and CLP cloud deployments; y-scope/clp#2439 is the deployment counterpart).Each plugin is selected by its own env var, since the coordinator JAR and worker
.soinstall into different locations (set either or both):CI builds the image per architecture on every run and assembles a multi-arch
:<version>tag, but pushes to GHCR only from the default branch and version tags. Local builds (task package) load the building machine's:<version>-<arch>tag. Details intools/build-packages/README.md.Validation performed
build-installer-init-image.sh --loadfrom a locally built package tarball; a run with bothCOORDINATOR_PLUGIN_INSTALL_PATHandWORKER_PLUGIN_INSTALL_PATHset installed the JAR and the.so+lib/into the right targets (file list matches the tarball exactly); a run with neither set exited with the usage error.--user "$(id -u):$(id -g)"): installs cleanly, files owned by the running user.