Skip to content

feat(packaging): Add a busybox init-container installer image as a fourth distribution channel. #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jackluo923 merged 10 commits into from
Aug 2, 2026
Merged

feat(packaging): Add a busybox init-container installer image as a fourth distribution channel. #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
124b2b1
feat(packaging): Add a busybox init-container installer image as a fo…
jackluo923 Jul 21, 2026
fac30a9
fix(packaging): Don't preserve source ownership when installing plugi…
jackluo923 Jul 31, 2026
65737b0
docs(build-packages): Document the installer image's CI tagging and p…
jackluo923 Jul 31, 2026
b57f9ec
Update tools/build-packages/build-installer-init-image.sh
jackluo923 Jul 31, 2026
5ae71ac
Update .github/workflows/build-packages.yaml
jackluo923 Jul 31, 2026
192de1c
Update tools/build-packages/README.md
jackluo923 Jul 31, 2026
2b50e41
style(build-packages): Rename die() to panic() in build-installer-ini…
jackluo923 Aug 1, 2026
801b5fb
tools: reject package versions unusable as Docker image tags instead …
jackluo923 Aug 1, 2026
6031345
tools: build the multi-arch manifest from digests; drop the local -<a…
jackluo923 Aug 1, 2026
4ec3ed2
docs(packaging): document tar prerequisite
junhaoliao Aug 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions .github/workflows/build-packages.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
# resolve-version use the input version or derive it from pom.xml
# build-dependency-image ensure the hash-tagged build-env image exists
# build build .deb / .rpm / .tar.gz artifacts per architecture
# image build the busybox installer image per arch (push on main/tags)
# manifest combine per-arch images into a multi-arch tag (main/tags only)
#
# See tools/build-packages/README.md for package-build details.

Expand Down Expand Up @@ -58,6 +60,7 @@ jobs:
runs-on: "ubuntu-24.04"
outputs:
version: "${{steps.version.outputs.version}}"
publish: "${{steps.publish.outputs.publish}}"
steps:
- name: "Check out repository"
uses: "actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0" # v7.0.0
Expand Down Expand Up @@ -88,6 +91,19 @@ jobs:
echo "version=${version}" >> "$GITHUB_OUTPUT"
echo "::notice::Package version: ${version}"

- name: "Decide whether to publish images"
id: "publish"
env:
DEFAULT_BRANCH: "${{github.event.repository.default_branch}}"
run: |-
# Publish images only from the default branch and version tags so feature
# branches don't overwrite shared tags.
if [[ ${GITHUB_REF_NAME} == "${DEFAULT_BRANCH}" || ${GITHUB_REF_TYPE} == tag ]]; then
echo "publish=true" >> "$GITHUB_OUTPUT"
else
echo "publish=false" >> "$GITHUB_OUTPUT"
fi

build-dependency-image:
uses: "./.github/workflows/build-dependency-image.yaml"
permissions:
Expand Down Expand Up @@ -182,3 +198,130 @@ jobs:
path: "packages/${{steps.packages.outputs.tar_gz_filename}}"
if-no-files-found: "error"
retention-days: 14

# Build the busybox installer image per architecture from the tarball artifact. This runs
# on the native runner (not inside the build-env container) so docker/buildx is available.
image:
needs: ["build", "resolve-version"]
permissions:
contents: "read"
packages: "write"
strategy:
fail-fast: false
matrix:
include:
- arch: "amd64"
runner: "ubuntu-24.04"
- arch: "arm64"
runner: "ubuntu-24.04-arm"
runs-on: "${{matrix.runner}}"
timeout-minutes: 30
env:
ARCH: "${{matrix.arch}}"
# Matrix legs share one output namespace, so each leg fills only its own arch's key
# (the other resolves to empty and is ignored by the merge below).
outputs:
digest_amd64: "${{steps.image.outputs.digest_amd64}}"
digest_arm64: "${{steps.image.outputs.digest_arm64}}"
steps:
- name: "Check out repository"
uses: "actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0" # v7.0.0
with:
persist-credentials: false

- name: "Download tarball artifact"
uses: "actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c" # v8.0.1
with:
pattern: "*-linux-${{matrix.arch}}.tar.gz"
path: "dist"
merge-multiple: true

- name: "Log in to GHCR"
if: "${{needs.resolve-version.outputs.publish == 'true'}}"
uses: "docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0" # v4.4.0
with:
registry: "ghcr.io"
username: "${{github.actor}}"
password: "${{secrets.GITHUB_TOKEN}}"

- name: "Set up Docker Buildx"
uses: "docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c" # v4.2.0

- name: "Build the installer image (and push when publishing)"
id: "image"
env:
PUBLISH: "${{needs.resolve-version.outputs.publish}}"
run: |-
image_repo="ghcr.io/$(printf '%s' "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]')"
shopt -s nullglob
tarballs=(dist/*-linux-"${ARCH}".tar.gz)
if (( ${#tarballs[@]} != 1 )); then
echo "::error::Expected exactly one tarball for ${ARCH}, found ${#tarballs[@]}"
exit 1
fi
# Push from the default branch / tags; otherwise just build to validate. When
# pushing, capture the image digest so the manifest job can combine the exact
# images this run built (per-arch tags are mutable and a concurrent run could
# move them between this job and the manifest job).
args=(--load)
if [[ "${PUBLISH}" == "true" ]]; then
args=(--push --digest-file digest.txt)
fi
bash tools/build-packages/build-installer-init-image.sh \
--tarball "${tarballs[0]}" \
--arch "${ARCH}" \
--repo "${image_repo}" \
"${args[@]}"
if [[ "${PUBLISH}" == "true" ]]; then
echo "digest_${ARCH}=$(cat digest.txt)" >> "$GITHUB_OUTPUT"
fi

# Combine the per-architecture installer images into one multi-arch version tag.
manifest:
needs: ["image", "resolve-version"]
if: "${{needs.resolve-version.outputs.publish == 'true'}}"
permissions:
packages: "write"
runs-on: "ubuntu-24.04"
timeout-minutes: 15
env:
VERSION: "${{needs.resolve-version.outputs.version}}"
DIGEST_AMD64: "${{needs.image.outputs.digest_amd64}}"
DIGEST_ARM64: "${{needs.image.outputs.digest_arm64}}"
steps:
- name: "Check out repository"
uses: "actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0" # v7.0.0
with:
persist-credentials: false

- name: "Log in to GHCR"
uses: "docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0" # v4.4.0
with:
registry: "ghcr.io"
username: "${{github.actor}}"
password: "${{secrets.GITHUB_TOKEN}}"

- name: "Create multi-arch manifest"
run: |-
image_repo="ghcr.io/$(printf '%s' "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]')"
# Validate the version with the same shared helper the build script uses, so the
# manifest tag always matches the per-arch tags (versions with characters Docker
# tags can't represent, like '+' or '~', fail loudly instead of being mangled).
source tools/build-packages/dependency-image/utils.sh
if ! tag_version="$(package_version_to_image_tag "${VERSION}")"; then
echo "::error::Version '${VERSION}' can't be used as an image tag"
exit 1
fi
for digest in "${DIGEST_AMD64}" "${DIGEST_ARM64}"; do
if [[ ! "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "::error::Missing or malformed image digest: '${digest}'"
exit 1
fi
done
# Reference the per-arch images by immutable digest rather than by tag: the
# per-arch tags are mutable, so a concurrent publish run could move them between
# the image jobs and this one, silently mixing images from different runs.
docker buildx imagetools create \

@junhaoliao junhaoliao Jul 31, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we consider this -

the architecture tags are mutable shared names, so overlapping runs can interleave before this manifest resolves them. publish run-scoped staging tags, then promote only those references after both matrix legs succeed:

# image job
staging_version="${VERSION}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
# pass --version "${staging_version}" and --push to the builder

# manifest job
source_tag="${tag_version}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
docker buildx imagetools create --prefer-index=false \
  --tag "${image_repo}:${tag_version}-amd64" "${image_repo}:${source_tag}-amd64"
docker buildx imagetools create --prefer-index=false \
  --tag "${image_repo}:${tag_version}-arm64" "${image_repo}:${source_tag}-arm64"
docker buildx imagetools create --tag "${image_repo}:${tag_version}" \
  "${image_repo}:${source_tag}-amd64" "${image_repo}:${source_tag}-arm64"

@jackluo923 jackluo923 Aug 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — the race is real: the per-arch tags are mutable shared names, so two overlapping publish runs (e.g. a re-run of an older attempt, or a tag build racing main) could produce a manifest mixing images from different runs.

Rather than run-scoped staging tags, I went with referencing the images by digest: each image leg captures the pushed image's registry digest (buildx --metadata-file) and exposes it as a job output, and the manifest job does imagetools create --tag : @ @. Same guarantee — the manifest combines exactly the images this run built — but:

  • digests are immutable, so there's no window at all (staging tags are still mutable names, just less likely to collide);
  • no staging tags accumulating in the GHCR package (they'd need a cleanup step to avoid piling up per run);
  • no extra pushes — the digest is a side effect of the push we already do.

The per-arch :- tags are still pushed for debugging convenience; they just no longer carry correctness. The final : tag stays mutable by design — that's the SNAPSHOT contract (each publish is supposed to move it); the fix only ensures each move is internally consistent.

--tag "${image_repo}:${tag_version}" \
"${image_repo}@${DIGEST_AMD64}" \
"${image_repo}@${DIGEST_ARM64}"
28 changes: 24 additions & 4 deletions tools/build-packages/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,17 @@
# CLP Presto connector packaging

This directory builds installable `.deb`, `.rpm`, and `.tar.gz` artifacts for the CLP
Presto connector (coordinator + worker) on `amd64` and `arm64`.
This directory builds installable `.deb`, `.rpm`, and `.tar.gz` artifacts, plus a busybox
init-container installer image, for the CLP Presto connector (coordinator + worker) on
`amd64` and `arm64`.

CI packaging runs `tools/build-packages/internal/container/build-artifacts.sh`
through `.github/workflows/build-packages.yaml`. Local builds use
`build-packages.sh`, which resolves the build-env image and invokes the same
container-side script.

Supported package version format: must start with a digit and use only
`[0-9A-Za-z.+~-]`.
`[0-9A-Za-z.+~-]`. The installer image additionally rejects versions containing
`+` or `~` (Docker tags can't represent them).

For command options, run `--help` on the relevant entry point.

Expand All @@ -25,6 +27,24 @@ task package

A thin wrapper over `./tools/build-packages/build-packages.sh` (call that directly if `go-task` isn't installed). Both accept `--output DIR`, `--version VER`, and `--with-ca-certs`; with the task, put `--` before the flags: `task package -- --output DIR`.

### Installer image

`task package` also builds and loads a busybox init-container image that bundles both plugins. Its entrypoint copies each component into a mounted volume named by `COORDINATOR_PLUGIN_INSTALL_PATH` / `WORKER_PLUGIN_INSTALL_PATH` (set either or both):

```bash
docker run --rm -e WORKER_PLUGIN_INSTALL_PATH=/plugins -v "$(pwd)/plugins:/plugins" \
ghcr.io/y-scope/clp-plugin-presto-connector:<version>
```

Run `./tools/build-packages/build-installer-init-image.sh --help` to build it standalone from any package tarball.

In CI, `build-packages.yaml` builds the image per architecture on every run and
combines them into a multi-arch `:<version>` tag; pushes to GHCR happen only
from the default branch and version tags. Local builds load the same
`:<version>` tag (single-arch, for the build host) — a locally-built image
therefore shadows the published one in your Docker daemon until you
`docker pull` it.

The build runs inside a hash-tagged **build-env image** (`env-<hash>`) based on
`manylinux_2_28`. `build-dependency-image.sh` resolves it from the local Docker
cache, this repository's GHCR package, or a local build, reusing the cached
Expand All @@ -42,7 +62,7 @@ while `packages/` is owned by the invoking user.

### Prerequisites

Docker with buildx (usable without `sudo`), git, `sha256sum` or `shasum`, and
Docker with buildx (usable without `sudo`), git, `tar`, `sha256sum` or `shasum`, and
~10 GB free disk for the build-env image.

## Target-CPU flags
Expand Down
18 changes: 0 additions & 18 deletions tools/build-packages/build-dependency-image.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,6 @@ set -o pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
source "${script_dir}/dependency-image/utils.sh"

# Derive this repo's GHCR namespace from its GitHub origin remote.
image_repo_from_origin() {
local remote_url owner_repo
remote_url="$(git -C "${_REPO_ROOT}" remote get-url origin)"
case "${remote_url}" in
https://github.com/*) owner_repo="${remote_url#https://github.com/}" ;;
git@github.com:*) owner_repo="${remote_url#git@github.com:}" ;;
ssh://git@github.com/*) owner_repo="${remote_url#ssh://git@github.com/}" ;;
*)
echo >&2 "ERROR: can't derive GHCR image repo from origin remote: ${remote_url}"
echo >&2 " Expected a github.com remote."
exit 1
;;
esac
owner_repo="${owner_repo%.git}"
printf 'ghcr.io/%s\n' "$(printf '%s' "${owner_repo}" | tr '[:upper:]' '[:lower:]')"
}

host_platform() {
case "$(uname -m)" in
x86_64) printf 'linux/amd64\n' ;;
Expand Down
Loading
Loading