Build and publish toolchain images in a single job - #899
Draft
hjiawei wants to merge 2 commits into
Draft
Conversation
A version bump used to run three full pipelines. The pull request built all
five images across every architecture, the branch push built them again and
published nothing, and the tag push built them a third time before each
promotion rebuilt the 13 images it was about to push. Seventy builds ran per
release-branch bump; thirteen produced an image anyone could pull.
Publishing rebuilt because the build and the push happened in different jobs
on different VMs. Split each image into a pull-request block that builds and
tests, and a branch block that builds, tests and pushes in one job. Exactly
one of the pair runs per event, so the push happens on the machine that
already holds the image and calico-*-cd no longer has calico-*-image as a
prerequisite.
Only the branch blocks carry the docker push secret, and they are gated on
`pull_request !~ '.*'` as well as the branch name. Semaphore's `branch`
resolves to a pull request's *target* branch, so `branch = 'master'` alone is
true on a pull request into master. The promotions this replaces hinted at
that: each was gated on `branch = 'master' OR tag =~ ...` and *also* wrapped
its command in `if [ -z "${SEMAPHORE_GIT_PR_NUMBER}" ]`, a second guard that
only makes sense if the first condition can be true on a pull request.
Gate every block with change_in, so a Kubernetes bump no longer rebuilds
base, binfmt, rust-build and tinygo. Let release branches publish their
branch tag the way master already does, and publish the release tag from the
same job by reading the tag create-tag-on-version-change.yml has created by
then, which leaves the tag pipeline with nothing to build.
The promotions are folded into the main pipeline: manifests need no local
image, and one pipeline is easier to follow than six.
That takes a release-branch bump from 70 builds to 8. Publishing also now
happens strictly after the tests pass, which the promotions did not
guarantee -- their auto_promote had no result = 'passed' condition.
Assemble semaphore.yml from fragments under .semaphore/semaphore.yml.d the
way projectcalico/calico does, so each image's blocks can be edited on their
own. Block fragments sit at column zero and the generator indents them, so
each one is valid YAML by itself. The numeric prefix groups blocks by what
the image is, not by build order; no toolchain image is built FROM another.
A new Checks block verifies the generated file matches its fragments and
runs shellcheck, and every image block depends on it, so a stale pipeline
file stops the builds before any emulated build starts.
Raise execution_time_limit to 120 minutes: a publish job now does the work
that the build pipeline and the publish promotion each had 60 minutes for,
and the ppc64le and s390x builds run under QEMU.
hjiawei
force-pushed
the
ci-build-once
branch
6 times, most recently
from
August 25, 2026 23:37
0cbaecb to
d3a71b9
Compare
After the previous commit a release-branch bump builds each image twice: once
in the pull request, once again on the merge. Nothing carries an image
between the two, because Semaphore jobs each start on a clean VM.
Cache every built image as a zstd tarball in GCS so a merge reuses what its
pull request built, which matters most for arm64, ppc64le and s390x: those
build under QEMU emulation.
An entry is keyed by two things, because the source tree alone does not
determine the image:
1. A hash of the committed files that can change that image, so a pull
request and the merge that follows it share an entry.
2. The digests of the base images it is built FROM. Those are floating tags,
and a new release means a rebuild would produce a different image. UBI is
rebuilt roughly daily; almalinux:9 and fedora:44 roughly quarterly.
That covers new base releases but not everything, because dnf and microdnf
pull from repositories that ship updates between base image retags --
almalinux:9 can sit unchanged for months while its repositories do not. A
72-hour expiry is the backstop for that drift, and it is also the staleness
bound on anything published, which before this cache was always built fresh.
A cache that cannot be reached never fails a build: a restore reports a miss
so the caller builds, and a store quietly does nothing. Forks cannot write to
the cache at all -- a fork could otherwise upload an image that does not match
the tree its build ID names, and a later merge of that innocuous-looking tree
would publish it.
Leave calico/base's publish ungated and on master only, as its promotion was.
base exists to carry current UBI packages: its Dockerfile barely changes, but
every rebuild picks up microdnf upgrade, and it was republished on every
master push, roughly every three days. Change-gating it would have cut that to
whenever the Dockerfile moved. Keying on the UBI digest means it rebuilds when
UBI does.
A restored image still needs its registry tags, so each image gains a -retag
target that applies them to whatever is in the local daemon, and the build
targets call it instead of inlining the retag. base builds every architecture
in one job because they share a single BRANCH_NAME, so its publish block loops
rather than using a matrix.
The bucket is this repo's own. calico-transient-build-artifacts-europe-west3
belongs to projectcalico/calico, where the images/ prefix holds a Docker Hub
fallback copy of the published calico/go-build keyed by GO_BUILD_VER -- a
consumer-side pull-through cache, not a place to put build output. Create
calico-toolchain in europe-west3, where the Semaphore runners are, and grant
it to the service account calico already uses.
Semaphore's own cache was measured and rejected: it caps at 9.6 GB per
project and one commit's images come to 8.7 GB compressed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
One version bump landing on a release branch runs 70 image builds, of which 13 produce an image anyone can pull. Three full pipelines run for the same commit: the pull request, the branch push, and the tag push that follows it ten seconds later.
Four things cause the repeats:
calico-*-cdlistscalico-*-imageas a prerequisite, and a promotion runs on a fresh VM, so publishing rebuilds all 13 images it is about to push.branch = 'master'nor the release-tag pattern, so all 19 of its builds are discarded.create-tag-on-version-change.ymltags the commit that was just pushed, so the tag pipeline builds an identical tree a second time.Change
Each image now has a pull-request block that builds and tests, and a branch block that builds, tests and pushes in one job. Exactly one of the pair runs per event, so the push happens on the machine that already holds the image and
calico-*-cdno longer needs its build prerequisite.Only the branch blocks carry the
dockerpush secret, and they are gated onpull_request !~ '.*'as well as the branch name. Semaphore'sbranchresolves to a pull request's target branch, so the branch name alone is true on a pull request into master.Every block is change-gated. Release branches publish their branch tag the way master already does, and the release tag is published from the same job by reading the tag that
create-tag-on-version-change.ymlhas created by then, which leaves the tag pipeline with nothing to build.The five promotions are folded into the main pipeline, and
semaphore.ymlis generated from fragments under.semaphore/semaphore.yml.d/, following the same pattern as projectcalico/calico.This takes a release-branch bump from 70 builds to 8. The second commit caches each built image as a zstd tarball keyed by a hash of its inputs, so a merge reuses what its pull request built, taking it to 4.
Notes for reviewers
calico-transient-toolchain-artifacts-europe-west3, writable by thegoogle-service-account-for-gceservice account. calico's own build-artifacts bucket is not suitable: itsimages/prefix is a Docker Hub fallback keyed byGO_BUILD_VER, not a place for build output.dnf upgradealso pulls from repositories that move between base retags, so entries additionally expire after 72 hours.calico/base's publish stays ungated and master-only, as its promotion was. It exists to carry current UBI packages and was republished every master push, roughly every three days; change-gating it would have cut that to whenever its Dockerfile moved.hack/image-cache.shrelies onSEMAPHORE_GIT_PR_SLUGandSEMAPHORE_GIT_REPO_SLUG.Draft while the pipeline runs against it.
🤖 Generated with Claude Code