Skip to content

fix: build multi-platform images by digest and merge manifests#447

Merged
fzipi merged 7 commits into
mainfrom
fix/multiarch-publish-digest-merge
Jul 5, 2026
Merged

fix: build multi-platform images by digest and merge manifests#447
fzipi merged 7 commits into
mainfrom
fix/multiarch-publish-digest-merge

Conversation

@fzipi

@fzipi fzipi commented Jul 4, 2026

Copy link
Copy Markdown
Member

Summary

Fixes #443 without reverting the native-ARM-runner speedup from 7c1554f (see #445 for the revert alternative).

  • Root cause: 7c1554f split each bake target's build into one job per platform to use native ARM runners, but every job still pushed under the same final tags. The three concurrent pushes for a target raced, and only the last platform to finish survived on Docker Hub/GHCR — this is why the LTS tags ended up single-arch.
  • Fix: each per-platform job now builds and pushes by digest only (push-by-digest=true, no tag) to one registry. A new merge job (one per target) downloads all of that target's platform digests and assembles them into a single multi-arch manifest with docker buildx imagetools create, pushing to both Docker Hub and GHCR in the same call (imagetools create copies cross-registry as needed).
  • Cosign signing moves to the merge job so it signs the final manifest-list digest, not a single-platform one.
  • docker-bake.hcl's versioned tags embed a timestamp (formatdate("YYYYMMDDHHMM", timestamp())), so tags are resolved once in prepare via docker buildx bake --print and shared to every job through an uploaded artifact — otherwise each job re-evaluating the bake file would compute different timestamps and the merge would target mismatched tags.

Validation

Since this touches the production release pipeline, I validated the core mechanics locally against two throwaway local registries (not in CI) before opening this:

  • docker buildx bake --print with --set target.tags= + --set target.output=type=image,name=...,push-by-digest=true,... resolves cleanly (tags fully cleared, single custom output).
  • Built the same synthetic image for linux/amd64 and linux/arm64 by digest against a local registry, then ran docker buildx imagetools create with two -t destinations across two separate local registries. Confirmed: the merge correctly flattened the per-platform attestation manifests into one clean multi-arch index, and the cross-registry copy to the registry with zero prior content succeeded, producing an identical manifest in both.
  • actionlint and zizmor both pass clean on the rewritten workflow (no template-injection or credential-persistence findings).

Test plan

  • Merge and trigger a real release; confirm each of the 12 published tags is multi-arch (docker buildx imagetools inspect <tag> shows all 3 platforms) on both Docker Hub and GHCR
  • Confirm cosign signatures verify against the final manifest-list digest
  • Confirm build time improvement vs. the pre-7c1554f baseline is retained (native ARM runners still used per platform)

AI Disclosure

Per the project's AI-Assisted Contribution Policy:

  1. AI tools used: Claude (Sonnet 5, via Claude Code)
  2. What was generated or assisted: Root-cause analysis of the single-arch tag bug (Images only available for single architecture. #443), and the design and implementation of the digest-then-merge rewrite of .github/workflows/publish.yml (the prepare/build/merge job split, freezing bake tags once via docker buildx bake --print to avoid timestamp drift, and the jq/imagetools create orchestration).
  3. Review performed: actionlint and zizmor run against the workflow (zero findings). Before opening this PR, the risky part of the design — clearing a bake target's tags in favor of a digest-only output, and cross-registry manifest assembly via docker buildx imagetools create — was validated end-to-end against two throwaway local Docker registries (not just reasoned about), confirming the merge correctly flattens per-platform provenance/attestation manifests and replicates cross-registry. I reviewed the final workflow logic and the local validation results myself before publishing.

Summary by CodeRabbit

  • Chores
    • Improved the release publishing process for container images.
    • Release artifacts are now built and published in stages, which should make multi-platform image releases more reliable.
    • Signing now happens on the final published image set, helping ensure the released image bundle is complete.

7c1554f split each bake target's build into one job per platform (to
use native ARM runners) but every job still pushed under the same
final tags, so the three concurrent pushes for a target raced and
only the last platform survived on Docker Hub/GHCR (#443).

Each per-platform job now pushes its image by digest only (no tag),
and a new merge job assembles the digests for a target into a single
multi-arch manifest with docker buildx imagetools create, which also
handles the cross-registry copy to both Docker Hub and GHCR. Cosign
signing moves to the merge job so it signs the final manifest-list
digest instead of a single-platform one.

Tags are resolved once in the prepare job (docker buildx bake
--print) and shared via an uploaded artifact, since docker-bake.hcl
embeds a timestamp in versioned tags that would otherwise diverge if
recomputed independently by each job.
@coderabbitai

coderabbitai Bot commented Jul 4, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@fzipi, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 41 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d199c397-d0fd-484b-a82e-747721d86ecc

📥 Commits

Reviewing files that changed from the base of the PR and between dff4f6b and 138c58d.

📒 Files selected for processing (1)
  • .github/workflows/publish.yml
📝 Walkthrough

Walkthrough

The release publishing workflow was restructured into a digest-based multi-arch flow: prepare generates bake metadata and a build matrix, build pushes each platform image by digest and uploads digests as artifacts, and merge assembles a signed multi-arch manifest list from those digests.

Changes

Publish Workflow Digest-Based Multi-Arch Build

Layer / File(s) Summary
Prepare job: bake metadata and matrix generation
.github/workflows/publish.yml
Runs docker buildx bake --print default to generate bake-metadata.json, parses it with jq into a combined build matrix and target list, and uploads the metadata as an artifact.
Build job: push-by-digest platform builds and digest export
.github/workflows/publish.yml
Consumes the platform-aware matrix, downloads bake-metadata, determines the push-by-digest repository, runs bake with tags cleared and provenance/SBOM enabled, then extracts and uploads per-platform digests; removes prior per-image cosign signing.
Merge job: manifest list assembly and signing
.github/workflows/publish.yml
Downloads per-target digests, creates/pushes a multi-arch manifest list via docker buildx imagetools create, retrieves the resulting digest, and signs it with cosign using GitHub OIDC.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Prepare
  participant Build
  participant Merge
  participant Registry

  Prepare->>Prepare: docker buildx bake --print default
  Prepare->>Registry: upload bake-metadata.json artifact
  Build->>Registry: download bake-metadata.json
  Build->>Registry: push platform image by digest
  Build->>Registry: upload per-target digest artifacts
  Merge->>Registry: download digest artifacts
  Merge->>Registry: docker buildx imagetools create (manifest list)
  Merge->>Registry: cosign sign manifest digest
Loading

Related issues: #443 — restores multi-architecture image availability by rebuilding and signing a combined manifest list instead of pushing single-platform images.

Suggested labels: ci, workflow, docker

Suggested reviewers: dune73, theseion

🐰 A rabbit bakes each arch by digest deep,
Then stitches manifests before it sleeps,
Signs the whole with cosign's glow,
So multi-arch images freely flow.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The workflow now rebuilds and publishes multi-arch manifest lists from per-platform digests, which directly addresses the single-architecture release issue.
Out of Scope Changes check ✅ Passed The changes stay focused on release-image publishing, digest handling, manifest assembly, and signing, with no clear unrelated additions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main workflow change to build by digest and merge multi-platform manifests.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/multiarch-publish-digest-merge

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@fzipi fzipi requested a review from theseion July 4, 2026 09:27
@fzipi fzipi added the enhancement New feature or request label Jul 4, 2026
Comment thread .github/workflows/publish.yml Outdated
Comment thread .github/workflows/publish.yml Outdated
Comment thread .github/workflows/publish.yml Outdated
Comment thread .github/workflows/publish.yml
Comment thread .github/workflows/publish.yml Outdated
Comment thread .github/workflows/publish.yml
Comment thread .github/workflows/publish.yml
Comment thread .github/workflows/publish.yml
fzipi and others added 6 commits July 4, 2026 15:19
Co-authored-by: Max Leske <250711+theseion@users.noreply.github.com>
Co-authored-by: Max Leske <250711+theseion@users.noreply.github.com>
Co-authored-by: Max Leske <250711+theseion@users.noreply.github.com>
Co-authored-by: Max Leske <250711+theseion@users.noreply.github.com>
Co-authored-by: Max Leske <250711+theseion@users.noreply.github.com>
@fzipi fzipi requested a review from theseion July 4, 2026 13:38
@fzipi fzipi merged commit d72150c into main Jul 5, 2026
38 checks passed
@fzipi fzipi deleted the fix/multiarch-publish-digest-merge branch July 5, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Images only available for single architecture.

2 participants