feat(connectors): ship connector plugins in the iggy-connect docker image - #3658
feat(connectors): ship connector plugins in the iggy-connect docker image#3658kriti-sc wants to merge 11 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3658 +/- ##
=============================================
- Coverage 75.55% 17.31% -58.24%
Complexity 969 969
=============================================
Files 1317 1315 -2
Lines 154895 133818 -21077
Branches 128520 107520 -21000
=============================================
- Hits 117027 23175 -93852
- Misses 34321 110128 +75807
+ Partials 3547 515 -3032
🚀 New features to boost your workflow:
|
|
/ready |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either Thank you for your contribution! |
| PLUGIN_FLAGS="$(sed 's/^/-p /' connector-plugins.txt | tr '\n' ' ')" && \ | ||
| if [ "$PROFILE" = "debug" ]; then PROFILE_DIR=debug; PROFILE_FLAG=""; else PROFILE_DIR=release; PROFILE_FLAG="--release"; fi && \ | ||
| cargo zigbuild --locked --target ${RUST_TARGET} -p iggy-connectors $PLUGIN_FLAGS $PROFILE_FLAG && \ | ||
| cp /app/target/${RUST_TARGET}/${PROFILE_DIR}/iggy-connectors /app/iggy-connectors && \ | ||
| mkdir -p /app/plugins && \ | ||
| while IFS= read -r name; do \ | ||
| [ -z "$name" ] && continue; \ | ||
| so_file="/app/target/${RUST_TARGET}/${PROFILE_DIR}/lib${name}.so"; \ | ||
| if [ ! -f "$so_file" ]; then echo "expected plugin artifact missing: $so_file" >&2 && exit 1; fi; \ | ||
| cp "$so_file" /app/plugins/; \ | ||
| done < connector-plugins.txt |
There was a problem hiding this comment.
The plugin compilation lives in the shared builder stage, so the slim flavor pays for it too.
runtime (slim) copies /app/iggy-connectors and /app/LICENSE-binary out of builder. Both of those are produced by RUN layers that sit at or after this one, so --target runtime still executes this cargo zigbuild -p iggy-connectors $PLUGIN_FLAGS (17 cdylibs) and the license RUN below that also generates LICENSE-binary-fat. BuildKit cannot skip a layer that an earlier needed artifact depends on.
Net effect: the -slim build goes from a runtime-binary-only compile to roughly the full fat build time (~20 min per the PR description), plus a cargo-about pass over 18 dependency closures, and then discards all of it. With the matrix at flavors x arches that is two extra full plugin compiles per publish run.
Suggestion: keep builder producing only the runtime binary and LICENSE-binary, then add a stage that only the fat image consumes:
FROM builder AS plugin-builder
# cargo zigbuild ... $PLUGIN_FLAGS, collect /app/plugins, generate LICENSE-binary-fat
FROM runtime AS runtime-fat
COPY --from=plugin-builder /app/plugins/ /usr/local/lib/
COPY --from=plugin-builder /app/LICENSE-binary-fat /usr/share/doc/iggy-connect/LICENSE-binary--target runtime then stops at builder and the slim image build is unchanged from today.
| with: | ||
| context: ${{ steps.ctx.outputs.context }} | ||
| file: ${{ steps.config.outputs.dockerfile }} | ||
| target: ${{ inputs.target }} |
There was a problem hiding this comment.
target now changes the build graph, but the buildx cache ref is still only component- and arch-scoped, so the two flavors fight over one cache entry.
The cache composition step above builds cache-to / cache-from as type=registry,ref=${img}:buildcache-${arch},mode=max (and the shared ${shared_deps}:buildcache-${arch}) with no flavor component. Since docker_matrix is now component x flavor x platform, the fat and slim jobs for the same arch run concurrently and both push mode=max to apache/iggy-connect:buildcache-amd64. The cache index is last-writer-wins, so each run one flavor's cache export is effectively lost, and cache-from may pull an index describing the other flavor's graph.
Layer integrity is fine (BuildKit verifies digests), but the fat build is the expensive one and is exactly the one that needs a warm cache. ${img}:latest in cache-from is also the fat image under the new tag scheme, which the slim build will now pull as inline cache.
Suggestion: thread the flavor into the cache ref, e.g. pass the flavor suffix into this action and use ${img}:buildcache${suffix}-${arch}, so each flavor keeps its own cache the same way it now keeps its own digest artifacts.
| if [[ ! -f "$so_file" ]]; then | ||
| echo "::error::expected connector plugin artifact missing: ${so_file}. The plugin crate built no cdylib, or its name does not match lib<crate>.so." >&2 | ||
| exit 1 |
There was a problem hiding this comment.
This turns a previously tolerated miss into a hard failure at the same time as the plugin set silently grows, so the first master push after merge is the first real exercise of the new set.
Old behaviour was [[ -f "$so_file" ]] && cp "$so_file" "${outdir}/", a soft skip. Combined with default: "" at line 49, the derived list goes from the 10 hardcoded crates to 17: clickhouse_sink, delta_sink, doris_sink, http_sink, influxdb_sink, influxdb_source, mongodb_sink are new here. Any of those that does not emit lib<crate>.so on x86_64-unknown-linux-gnu / aarch64-unknown-linux-gnu now fails the edge release rather than being skipped.
The hard failure is the right call, the concern is that it is unverified for the 7 additions. Per the PR description the local validation was the docker image build, which is a different job and a different build graph. timeout-minutes: 60 is also unchanged for what is now a 17-crate release build on the standard runners, and the tarball grows accordingly (delta/arrow, iceberg, mongodb, surrealdb closures).
Worth running the Connector plugins job on this branch before merge to confirm all 17 produce a cdylib on both targets and that it fits in the timeout.
Closes #3614
Rationale
Dynamically inject plugins to the
apache/iggy-connectimage to reduce friction for testing out connectors.What changed?
This change publishes the image in two flavors:
Fat (edge, x.y.z, latest): the runtime plus every connector plugin baked into /usr/local/lib, which is already on the runtime's plugin search path. A config can load a bundled plugin by file name with no path. This is the new default.
Slim (edge-slim, x.y.z-slim, latest-slim): the runtime binary only, i.e. today's image, for deployments that bring their own plugins.
To this end:
Local Execution
AI Usage