CI: pin zenoh-flat-jni in a lockfile the org's bot can move, and resolve it in the build - #521
Conversation
CI tracked zenoh-flat-jni's default branch, which never went stale and was never reproducible: a CI result here stopped being determined by this repository's commit. Record the commit under test in a Cargo.lock instead, through a pin crate at the root that compiles to nothing anyone ships, and let settings.gradle.kts decide where zenoh-flat-jni source comes from - a directory given by -PflatJniDir, a path= in Cargo.toml, or the pinned commit fetched on demand with -PuseLocalFlatJni. A release takes none of those and resolves the Maven artifact; build.gradle.kts fails any publish task while a composite build is present. A lockfile is the pin eclipse-zenoh/ci's lockfile sync already knows how to move, so this repository stays an ordinary dependant of it. The workflow loses its zenoh-flat-jni checkout: the build fetches its own bindings, which is what makes 'gradlew jvmTest -PuseLocalFlatJni=true' reproduce a CI run anywhere.
Pointing a build at zenoh-flat-jni source is how you develop against it, not how you publish. PUBLISHING.md keeps only the part that is a release constraint: a composite build must never reach a publication.
The README answers how to work on this project, including across the stack: how to point a build at zenoh-flat-jni source, and how to reproduce a CI run. Which commit that is, and the lockfile sync that keeps it current, is CI's business - and it does not touch publishing, which resolves a release from Maven Central.
The README now answers what a user asks: use the published bindings (default, no Rust), build them from source as Cargo.toml says, or point at a checkout. Everything behind that - the pin crate, the fetch, the lockfile sync - moved to CI.md, which is where it belongs. -PuseLocalFlatJni now follows the manifest rather than only the lockfile: a git dependency with no Cargo.lock resolves one with Cargo instead of failing, so a fresh tree works the way a Rust developer expects. -PflatJniDir accepts an absolute path.
Also localJniCommit, so the three share one prefix.
Review on the zenoh-kotlin twin caught that the lockfile sync only fires when zenoh's Cargo.lock moves, so a zenoh-flat-jni-only fix would leave this pin sitting still - reproducible, but not current. update-flat-jni.yml runs cargo update -p zenoh-flat-jni on a schedule and opens the same kind of auto-merging pull request, gated on this repository's CI. Also: the pin crate pointed at PUBLISHING.md instead of CI.md, checkoutPinned hard-failed on a half-initialised checkout and refetched for an abbreviated commit, and the test section did not mention the two source options.
Verified rather than assumed: zenoh's Cargo.lock carries no zenoh-flat-jni entry, so overwriting this lockfile with it removes the pin, and rectifying re-resolves it to zenoh-flat-jni's main tip. zenoh's lockfile moves every day or two, and the resulting PR is gated on this repository's CI, so a zenoh-flat-jni commit that breaks these tests cannot land - detection and gating are already there without a second bot.
On the staleness objection: no new automation, and here is the measurementReview on the zenoh-kotlin twin (eclipse-zenoh/zenoh-kotlin#700) raised a blocking point: the pin is only as good as the bot that I first answered by adding one: a scheduled What I measuredOn this repository's actual lockfile, simulating what a sync run does:
So a sync run does not preserve a stale pin: it re-resolves it to whatever And the trigger fires far more often than I assumed. Why that settles itThe sync's pull request auto-merges only if this repository's CI passes, and
Detection and gating both already exist. A dedicated bot would have bought two What I did insteadDocumentation, where a claim was doing work it had not earned. The other review pointsAll fixed in 4f86578:
Ordering is unchanged: eclipse-zenoh/ci#465 lands first — it now carries both the |
The prefix compare only avoided a refetch when an abbreviation happened to match a cached HEAD; on a cold or different checkout it reached 'git fetch <url> <abbrev>', which no remote can answer. An empty value was accepted outright. Validate 40 hex digits before anything is fetched.
The rehearsal table and the test section were updated by renaming the property rather than by saying what it now does - it fetches the pinned commit, not a sibling checkout - and the test section did not mention that both source options need a Rust toolchain.
1 participant
CI here has to answer one question: which
zenoh-flat-jnidoes this SDK buildand test against? Both previous answers were bad in opposite ways. A hand-edited
SHA in
ci.ymlwas reproducible and went stale, because moving it was somebody'schore. #520 replaced it with zenoh-flat-jni's
main, which is never stale and isnot reproducible — a CI result stopped being determined by this repository's
commit.
A lockfile never makes that trade, because a bot moves it. So this makes the pin
be a lockfile, and then lets the build — rather than the workflow — act on it.
The pin: a crate at the root
Cargo.toml,ci/pin.rs,rust-toolchain.toml,Cargo.lock. The cratecompiles to nothing anyone ships; its entire content is one dependency:
whose only job is to turn the commit under test into a resolved lockfile entry:
Nothing builds this crate.
cargo buildat the root would compile zenoh toproduce an empty library — Gradle only reads these two files.
Why a lockfile: the eclipse-zenoh/ci part
Because
eclipse-zenoh/ci'ssync-lockfilesalready moves lockfiles, for everyzenoh dependant, whenever zenoh’s own
Cargo.lockmoves onmain, and thisrepository can just be one of them. Its
sequence, applied here:
Cargo.lockwith zenoh's — which carries nozenoh-flat-jnientry, so the pin is removed;
cargo clippy --all-targets --all-features -- --deny warnings) — which writes back zenoh-flat-jni'scurrent
maincommit, and proves it compiles, so a broken combination neverbecomes a pull request;
create-pull-request+gh pr merge --squash --auto, gated on thisrepository's own CI — which tests against the very commit that PR just wrote.
The pin makes a run reproducible from this repository's commit; the bot keeps it
from going stale. Neither property is new machinery — that is the point.
One related PR:
zenoh-flat/zenoh-flat-jnito the sync matrixand deletes the crate-path step, which exists for exactly two repositories —
zenoh-(java|kotlin) → zenoh-jni. That crate is gone from this repository, sothe sync here is currently broken; a manifest at the root is what fixes
it. This is also why the pin crate is at the root and not in a subdirectory: a
subdirectory would be asking to stay a special case.
The other sync special cases don't apply: the clippy-features one is for the DDS
plugins, and the extra
build-resources/opaque-typessteps are zenoh-c's secondmanifest.
One consequence worth naming: the sync's
cargo update zenohstep re-pinszenohin this lockfile, where zenoh is only a transitive dependency of the pincrate. Harmless — nothing is built from this lockfile — but the zenoh revision
recorded here is not necessarily the one inside the zenoh-flat-jni commit it
pins. The commit is the pin; the rest of the lockfile is a by-product.
Rehearsed the whole sequence locally against zenoh's real lockfile before
committing to it: the pin survives the round trip and comes back pointing at
main.Acting on the pin: in the build, not the workflow
settings.gradle.ktsdecides wherezenoh-flat-jnisource comes from, ifanywhere, in this order:
-PlocalJniDir=<path>path = "…"inCargo.toml-PuseLocalJni=trueCargo.lockpins, fetched into.zenoh-flat-jni/Row 2 is the reason
Cargo.tomlis parsed and not justCargo.lock: pointing abuild at your own checkout is then the ordinary Cargo edit, in the tracked file
where a Rust developer would look for it, with no Gradle properties involved. Row
3 fetches only when
.zenoh-flat-jni/is not already at the pinned commit, and-PlocalJniCommit=<sha>tries a different commit without touching the lockfile.A release takes row 4, and must — with a composite build the published
artifact would be built from source on the builder's disk while the POM still
claimed the released version. Nothing opts in by default, and
build.gradle.ktsnow fails any
publish*task while an included build is present, which catches aleftover property or a
path =left in the manifest.What that does to ci.yml
It removes work rather than moving it: -34 lines, five steps left, and not one
of them names a commit or runs Cargo.
The Gradle build fetches its own pinned bindings, and their test task depends on
their native build, so Gradle drives both git and cargo. Which means the second
line reproduces a full CI run anywhere — that is the property the redesign was
for. Formatting, clippy, the feature-leak test and the native build were also
deleted here: they are zenoh-flat-jni's own CI, on three platforms, for the very
commit pinned here, and re-running them from this repository only added ways for
two toolchains to disagree.
The identical change in eclipse-zenoh/zenoh-kotlin#700 is green on ubuntu and
macos with no checkout step, which is the proof that the fetch path works end to
end; the same run here locally passes 111/112, the one failure being a LAN peer
discovery test unrelated to this.
Moving the pin by hand
Normally you don't; the bot's PR does. When you need to:
CI.mddocuments this along with the two ways to defeat the mechanismrather than steer it:
rev = "…"in the manifest freezes resolution so the synccan no longer move the pin, and a committed
path = "…"leaves the lockfilepinning no commit at all (CI says so and fails, rather than checking out
something arbitrary).
Afterwards
When
org.eclipse.zenoh:zenoh-flat-jniis on Maven Central, all of this retiresin favour of
zenohFlatJniVersioningradle.properties— one pin for CI, localbuilds and releases, with no fetch, no Rust toolchain, and no pin crate.