Skip to content

ci(integration-tests): Add GitHub workflow job to run CLP-package integration tests on every push. #2407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
quinntaylormitchell wants to merge 8 commits into
base: main
Choose a base branch
from
Open

ci(integration-tests): Add GitHub workflow job to run CLP-package integration tests on every push. #2407

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
faba7ae
Move package tests over from PR 2334.
quinntaylormitchell Jul 21, 2026
bebd263
Use image reference instead of image ID.
quinntaylormitchell Jul 29, 2026
771feac
Fix the image ref thing
quinntaylormitchell Jul 29, 2026
3a2b518
Make sure package tests run on main; rename to package-integration tests
quinntaylormitchell Jul 30, 2026
5a9d76c
Add back the 'if:' statements
quinntaylormitchell Jul 30, 2026
0bd423e
Update docs.
quinntaylormitchell Jul 30, 2026
116a4f0
Merge branch 'main' into feature branch
quinntaylormitchell Aug 4, 2026
4fa0ac1
Comments from 2334.
quinntaylormitchell Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 40 additions & 3 deletions ...tions/clp-build-runtime-image/action.yaml → ...tions/clp-build-package-image/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: "clp-build-runtime-image"
name: "clp-build-package-image"
description: "Builds a container image to be used for running CLP."

inputs:
Expand All @@ -22,6 +22,14 @@ inputs:
description: >-
Platform version codename (e.g. jammy, noble) used as the runtime image base version.
required: true
push_image:
description: "Whether to publish the package image"
required: true

outputs:
image_ref:
description: "The tagged reference of the built container image."
value: "${{steps.compute-image-ref.outputs.IMAGE_REF}}"

runs:
using: "composite"
Expand All @@ -30,6 +38,7 @@ runs:
uses: "docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2"

- name: "Login to Image Registry"
if: "inputs.push_image == 'true'"
uses: "docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121" # v4.1.0
with:
registry: "${{inputs.image_registry}}"
Expand Down Expand Up @@ -65,6 +74,13 @@ runs:
flavor: |
suffix=-${{inputs.arch}}

- name: "Compute image reference"
id: "compute-image-ref"
shell: "bash"
run: |
echo "IMAGE_REF=$(head -n 1 <<< '${{steps.extract-gh-meta.outputs.tags}}')" \
>> "$GITHUB_OUTPUT"

- name: "Map arch to Docker platform"
id: "map-arch"
shell: "bash"
Expand All @@ -78,8 +94,21 @@ runs:
exit 1
fi

- name: "Compute image output properties"
id: "output-properties"
shell: "bash"
run: |
image_name="${{steps.compute-meta.outputs.IMAGE_NAME}}"
if [[ "${{inputs.push_image}}" == "true" ]]; then
echo "image_output=type=registry" >> "$GITHUB_OUTPUT"
else
image_path="/tmp/${image_name}-image-${{inputs.arch}}.tar"
echo "image_path=${image_path}" >> "$GITHUB_OUTPUT"
echo "image_output=type=docker,dest=${image_path}" >> "$GITHUB_OUTPUT"
fi

- name: "Build and Push"
if: "github.event_name != 'pull_request' && github.ref == 'refs/heads/main'"
id: "build-and-push"
Comment on lines +97 to +111

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Hardcoded /tmp tar path may collide across concurrent jobs on shared self-hosted runners.

image_path="/tmp/${image_name}-image-${{inputs.arch}}.tar" is fixed per arch but not per-workflow-run. On the self-hosted runner pool used by this repo, two concurrent runs (e.g., two PRs) building the same arch could write to the same /tmp path simultaneously.

🛠️ Proposed fix: scope the path to the run 
-          image_path="/tmp/${image_name}-image-${{inputs.arch}}.tar"
+          image_path="${RUNNER_TEMP}/${image_name}-image-${{inputs.arch}}-${{github.run_id}}.tar"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: "Compute image output properties"
id: "output-properties"
shell: "bash"
run: |
image_name="${{steps.compute-meta.outputs.IMAGE_NAME}}"
if [[ "${{inputs.push_image}}" == "true" ]]; then
echo "image_output=type=registry" >> "$GITHUB_OUTPUT"
else
image_path="/tmp/${image_name}-image-${{inputs.arch}}.tar"
echo "image_path=${image_path}" >> "$GITHUB_OUTPUT"
echo "image_output=type=docker,dest=${image_path}" >> "$GITHUB_OUTPUT"
fi
- name: "Build and Push"
if: "github.event_name != 'pull_request' && github.ref == 'refs/heads/main'"
id: "build-and-push"
- name: "Compute image output properties"
id: "output-properties"
shell: "bash"
run: |
image_name="${{steps.compute-meta.outputs.IMAGE_NAME}}"
if [[ "${{inputs.push_image}}" == "true" ]]; then
echo "image_output=type=registry" >> "$GITHUB_OUTPUT"
else
image_path="${RUNNER_TEMP}/${image_name}-image-${{inputs.arch}}-${{github.run_id}}.tar"
echo "image_path=${image_path}" >> "$GITHUB_OUTPUT"
echo "image_output=type=docker,dest=${image_path}" >> "$GITHUB_OUTPUT"
fi
- name: "Build and Push"
id: "build-and-push"
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/actions/clp-build-package-image/action.yaml around lines 97 - 111,
Update the image_path construction in the “Compute image output properties” step
to include a workflow-run-unique identifier, such as github.run_id, alongside
image_name and arch. Keep the existing output properties and push_image
branching unchanged while ensuring concurrent runs cannot write the same tar
path.

uses: "docker/build-push-action@471d1dc4e07e5cdedd4c2171150001c434f0b7a4"
with:
context: "./"
Expand All @@ -88,8 +117,16 @@ runs:
# Disable provenance to create a simple image instead of a manifest list.
# This allows `docker manifest create` to combine the per-arch images later.
provenance: false
push: true
outputs: "${{steps.output-properties.outputs.image_output}}"
tags: "${{steps.extract-gh-meta.outputs.tags}}"
labels: "${{steps.extract-gh-meta.outputs.labels}}"
build-args: |
UBUNTU_VERSION_CODENAME=${{inputs.platform_version_codename}}

- name: "Upload package image as artifact"
if: "inputs.push_image == 'false'"
uses: "actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02"
with:
name: "clp-package-image-${{inputs.arch}}"
path: "${{steps.output-properties.outputs.image_path}}"
retention-days: 1
Comment on lines +126 to +132

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== workflow/action files =="
git ls-files '.github/workflows/*.yaml' '.github/actions/**/action.yaml' | sed -n '1,200p'

echo
echo "== target usage =="
rg -n "actions/upload-artifact|node-version|defaults:|runs-on:|actions/checkout|clp-build-package-image|path_to_repo" .github -S || true

echo
echo "== target snippet =="
nl -ba .github/actions/clp-build-package-image/action.yaml | sed -n '1,170p'

echo
echo "== tag/SHA resolution attempts =="
for ref in actions/upload-artifact@v4.6.2 actions/upload-artifact@v6.0.0; do
  echo "--- $ref"
  gh api "repos/$ref" --jq '{ref: .ref, object_type: .object.type, tag_object_sha: .object.sha, target_commit: (if .object.type=="tag" then .object.target_sha else .object.sha end)}' 2>/dev/null || echo "gh api failed"
done

Repository: y-scope/clp

Length of output: 7426

🌐 Web query:

GitHub Actions upload-artifact Node.js tool cache Node 20 deprecation Node 24 runners June 2026

💡 Result:

As of July 30, 2026, GitHub Actions has transitioned to Node.js 24 as the default runtime for JavaScript/TypeScript actions, a change that took effect on June 16, 2026 [1][2]. Key details regarding this transition include: Deprecation of Node 20: Node.js 20 reached end-of-life (EOL) in April 2026, leading to its deprecation as a supported runtime for GitHub Actions [1]. Default Runner Version: Beginning June 16, 2026, GitHub-hosted runners default to Node.js 24 [1][2]. Workflows using actions that still target Node.js 20 may trigger deprecation warnings [3][2]. Temporary Opt-Out: While Node.js 24 is the new default, users can temporarily continue using Node.js 20 by setting the environment variable ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true in their workflow file or on their runner machine [1][4]. This is a temporary measure intended only until Node.js 20 is completely removed from the runners, which is scheduled for the fall of 2026 [1]. Action Updates (upload-artifact): The official actions/upload-artifact action has been updated to v6.0.0, which uses Node.js 24 by default [5][6]. Users are strongly encouraged to update to the latest versions of their actions to ensure compatibility with the new Node.js 24 runtime [1]. Self-Hosted Considerations: Self-hosted runners require an upgrade to at least version 2.327.1 to support the Node.js 24 runtime [5][6]. Additionally, Node.js 24 does not support ARM32, meaning self-hosted runners on that architecture are no longer supported [1]. For users on GitHub Enterprise Server (GHES), check documentation as some versions may still require older, Node-specific releases of the upload-artifact action [7][8].

Citations:

🌐 Web query:

actions/upload-artifact v4 node-version action.yml

💡 Result:

The actions/upload-artifact@v4 action does not accept a node-version input in its action.yml file [1][2]. In GitHub Actions, the node-version input is specific to the actions/setup-node action, which is used to configure the Node.js environment on the runner [3][4]. The actions/upload-artifact action is a JavaScript-based GitHub Action that internally specifies its required Node.js runtime environment using the runs.using field within its own action.yml (e.g., node20 or similar, depending on the specific patch version of v4) [4][5]. Users do not need to—and cannot—configure the Node.js version of the artifact action itself through its inputs [1][6]. If you need to use a specific version of Node.js for tasks within your workflow, you should use actions/setup-node@v4 in a preceding step [3][4].

Citations:

Align upload-artifact to the Node 24-based release. The pinned release is still Node 20-based, while GitHub-hosted runners now default to Node 24; use the same actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f / v6.0.0 used by the artifact build workflow.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/actions/clp-build-package-image/action.yaml around lines 126 - 132,
Update the “Upload package image as artifact” step to use
actions/upload-artifact pinned to b7c566a772e6b6bfb58ed0dc250532a479d7789f
(v6.0.0), matching the artifact build workflow, while preserving its existing
condition and inputs.

146 changes: 138 additions & 8 deletions .github/workflows/clp-artifact-build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ on:
env:
DEPS_IMAGE_NAME_PREFIX_AARCH64: "clp-core-dependencies-aarch64-"
BINARIES_ARTIFACT_NAME_PREFIX: "clp-core-binaries-"
PACKAGE_ARTIFACT_NAME_PREFIX: "clp-package-"
PACKAGE_IMAGE_ARTIFACT_NAME_PREFIX: "clp-package-image-"
PYTHON_WHEELS_ARTIFACT_NAME_PREFIX: "clp-python-wheels-"
DEPS_IMAGE_NAME_PREFIX_X86: "clp-core-dependencies-x86-"
# Bare (no arch) prefix for multi-arch dependency images; the arch is carried in the tag
Expand Down Expand Up @@ -724,8 +726,8 @@ jobs:
'build/lint-clang-tidy/**/*'
)}}

package-image:
name: "package-image-${{matrix.arch}}"
package:
name: "package-${{matrix.arch}}"
# NOTE: The aarch64 job is skipped on PRs, so we accept both 'success' and 'skipped'.
if: >-
!cancelled() && !failure() && (
Expand All @@ -740,11 +742,11 @@ jobs:
- "ubuntu-jammy-aarch64-deps-image"
strategy:
matrix:
# Only build arm64 images on push to main to save CI resources on PRs.
arch: >-
# arm64 packages are only built on main to save CI resources on PRs.
arch: &package_matrix_arch >-
${{github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
&& fromJSON('["amd64", "arm64"]') || fromJSON('["amd64"]')}}
runs-on: >-
runs-on: &package_arch_runner >-
${{
matrix.arch == 'amd64'
&& (github.repository_owner == 'y-scope'
Expand Down Expand Up @@ -780,21 +782,149 @@ jobs:
HOME=/tmp
task package-build-deps

- uses: "./.github/actions/clp-build-runtime-image"
- id: "build-package-image"
uses: "./.github/actions/clp-build-package-image"
with:
image_registry: "ghcr.io"
image_registry_username: "${{github.actor}}"
image_registry_password: "${{secrets.GITHUB_TOKEN}}"
arch: "${{matrix.arch}}"
platform_version_codename: "jammy"
push_image: &package_image_published >-
${{github.event_name != 'pull_request' && github.ref == 'refs/heads/main'}}

- name: "Assemble the package directory structure"
uses: "./.github/actions/run-on-image"
env:
OS_NAME: "ubuntu-jammy"
with:
image_name: >-
${{format('{0}ubuntu-jammy',
matrix.arch == 'amd64' && env.DEPS_IMAGE_NAME_PREFIX_X86
|| env.DEPS_IMAGE_NAME_PREFIX_AARCH64)}}
use_published_image: >-
${{needs.filter-relevant-changes.outputs.ubuntu_jammy_image_changed == 'false'
|| (github.event_name != 'pull_request' && github.ref == 'refs/heads/main')}}
run_command: >-
CLP_CPP_MAX_PARALLELISM_PER_BUILD_TASK=$(getconf _NPROCESSORS_ONLN)
HOME=/tmp
task package-assemble OUTPUT_DIR=build/clp-package

- name: "Write the package image reference"
shell: "bash"
run: >-
echo "${{steps.build-package-image.outputs.image_ref}}"
> "build/clp-package/clp-package-image.id"

- name: "Tar the package directories"
shell: "bash"
run: >-
tar --create --file "build/clp-package.tar"
--directory "build/clp-package" .

- name: "Upload the package tarball artifact"
uses: "actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f" # v6.0.0
with:
name: "${{env.PACKAGE_ARTIFACT_NAME_PREFIX}}${{matrix.arch}}"
path: "build/clp-package.tar"
retention-days: 1
Comment on lines +796 to +830

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'Repository files matching task/workflow/package names:\n'
git ls-files | rg '(^|/)(taskfile.*|.*workflow.*|.*package.*|.*install.*|.*release.*)' || true

printf '\nRelevant workflow section:\n'
if [ -f .github/workflows/clp-artifact-build.yaml ]; then
  nl -ba .github/workflows/clp-artifact-build.yaml | sed -n '700,790p'
fi

printf '\nSearch for package task/package-assemble/version behavior:\n'
rg -n "task package|package-assemble|G_PACKAGE_VERSION|VERSION|checksum|clp-package-image|clp-package.tar" -S .

Repository: y-scope/clp

Length of output: 31017

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'taskfile.yaml task package/assemble excerpts:\n'
awk 'NR>=120 && NR<=165 {printf "%d\t%s\n", NR, $0}' taskfile.yaml

printf '\nWorkflow assembly/tar/upload excerpt:\n'
awk 'NR>=720 && NR<=785 {printf "%d\t%s\n", NR, $0}' .github/workflows/clp-artifact-build.yaml

printf '\nVERSION/CLP_PACKAGE_VERSION references in taskfile/docker/package docs:\n'
rg -n "G_PACKAGE_VERSION|CLP_PACKAGE_VERSION|package-assemble|task package|VERSION|checksum" taskfile.yaml Dockerfile components package templates docs 2>/dev/null || true

printf '\nRead-only model of taskfile dependency chain:\n'
python3 - <<'PY'
from pathlib import Path
p=Path("taskfile.yaml")
text=p.read_text()
for alias in ["package:", "package-assemble:"]:
    idx=text.find(alias)
    print(f"{alias} line={text.count(chr(10),0,idx)+1 if idx!=-1 else None}")
    if idx!=-1:
        print(text[idx:idx+450])
        print("---")
PY

Repository: y-scope/clp

Length of output: 21392

Write VERSION before tarring the CI package artifact.

task package runs package-assemble, rsyncs clp-package-image.id, writes VERSION, and then computes the checksum. This workflow invokes package-assemble directly and tarball/uses the image id immediately after, so the uploaded artifact omits the version file that the local full package contract includes.

🩹 Proposed fix: write VERSION alongside the image id 
       - name: "Write the package image reference"
         shell: "bash"
         run: >-
           echo "${{steps.build-package-image.outputs.image_ref}}"
           > "build/clp-package/clp-package-image.id"

+      - name: "Write the package version"
+        shell: "bash"
+        run: >-
+          echo "${{env.G_PACKAGE_VERSION}}" > "build/clp-package/VERSION"
+
       - name: "Tar the package directories"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: "Assemble the package directory structure"
uses: "./.github/actions/run-on-image"
env:
OS_NAME: "ubuntu-jammy"
with:
image_name: >-
${{format('{0}ubuntu-jammy',
matrix.arch == 'amd64' && env.DEPS_IMAGE_NAME_PREFIX_X86
|| env.DEPS_IMAGE_NAME_PREFIX_AARCH64)}}
use_published_image: >-
${{needs.filter-relevant-changes.outputs.ubuntu_jammy_image_changed == 'false'
|| (github.event_name != 'pull_request' && github.ref == 'refs/heads/main')}}
run_command: >-
CLP_CPP_MAX_PARALLELISM_PER_BUILD_TASK=$(getconf _NPROCESSORS_ONLN)
HOME=/tmp
task package-assemble OUTPUT_DIR=build/clp-package
- name: "Write the package image reference"
shell: "bash"
run: >-
echo "${{steps.build-package-image.outputs.image_ref}}"
> "build/clp-package/clp-package-image.id"
- name: "Tar the package directories"
shell: "bash"
run: >-
tar --create --file "build/clp-package.tar"
--directory "build/clp-package" .
- name: "Upload the package tarball artifact"
uses: "actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f" # v6.0.0
with:
name: "${{env.PACKAGE_ARTIFACT_NAME_PREFIX}}${{matrix.arch}}"
path: "build/clp-package.tar"
retention-days: 1
- name: "Assemble the package directory structure"
uses: "./.github/actions/run-on-image"
env:
OS_NAME: "ubuntu-jammy"
with:
image_name: >-
${{format('{0}ubuntu-jammy',
matrix.arch == 'amd64' && env.DEPS_IMAGE_NAME_PREFIX_X86
|| env.DEPS_IMAGE_NAME_PREFIX_AARCH64)}}
use_published_image: >-
${{needs.filter-relevant-changes.outputs.ubuntu_jammy_image_changed == 'false'
|| (github.event_name != 'pull_request' && github.ref == 'refs/heads/main')}}
run_command: >-
CLP_CPP_MAX_PARALLELISM_PER_BUILD_TASK=$(getconf _NPROCESSORS_ONLN)
HOME=/tmp
task package-assemble OUTPUT_DIR=build/clp-package
- name: "Write the package image reference"
shell: "bash"
run: >-
echo "${{steps.build-package-image.outputs.image_ref}}"
> "build/clp-package/clp-package-image.id"
- name: "Write the package version"
shell: "bash"
run: >-
echo "${{env.G_PACKAGE_VERSION}}" > "build/clp-package/VERSION"
- name: "Tar the package directories"
shell: "bash"
run: >-
tar --create --file "build/clp-package.tar"
--directory "build/clp-package" .
- name: "Upload the package tarball artifact"
uses: "actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f" # v6.0.0
with:
name: "${{env.PACKAGE_ARTIFACT_NAME_PREFIX}}${{matrix.arch}}"
path: "build/clp-package.tar"
retention-days: 1
🧰 Tools
🪛 zizmor (1.28.0)

[warning] 682-783: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block

(excessive-permissions)

[info] 769-769: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/clp-artifact-build.yaml around lines 749 - 783, Update the
package assembly steps around “Write the package image reference” to also create
the package’s VERSION file before “Tar the package directories” runs. Reuse the
same VERSION value and format established by the package-assemble/package flow,
ensuring build/clp-package contains VERSION alongside clp-package-image.id in
the uploaded artifact.


package-integration-tests:
name: "package-integration-tests-${{matrix.arch}}"
if: "!cancelled() && !failure() && needs.package.result == 'success'"
needs:
- "package"
strategy:
matrix:
# arm64 packages are only tested on main to save CI resources on PRs.
arch: *package_matrix_arch
runs-on: *package_arch_runner
env:
# `package` publishes the image to the registry on pushes to main and scheduled runs.
PACKAGE_IMAGE_PUBLISHED: *package_image_published
BUILD_DIR: "build"
steps:
- uses: "actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd" # v6.0.2
with:
submodules: "recursive"
Comment thread
quinntaylormitchell marked this conversation as resolved.
Outdated
persist-credentials: false

- name: "Download the package artifact"
uses: "actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e"
with:
name: "${{env.PACKAGE_ARTIFACT_NAME_PREFIX}}${{matrix.arch}}"
path: "${{env.BUILD_DIR}}/clp-package"

- name: "Untar the package"
working-directory: "${{env.BUILD_DIR}}/clp-package"
run: |-
tar xf clp-package.tar
rm clp-package.tar

- if: "env.PACKAGE_IMAGE_PUBLISHED == 'false'"
name: "Download the package image artifact"
uses: "actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e"
with:
name: "${{env.PACKAGE_IMAGE_ARTIFACT_NAME_PREFIX}}${{matrix.arch}}"
path: "${{env.BUILD_DIR}}/clp-package-image"

- if: "env.PACKAGE_IMAGE_PUBLISHED == 'false'"
name: "Load the package image"
shell: "bash"
env:
PACKAGE_IMAGE_DIR: "${{env.BUILD_DIR}}/clp-package-image"
PACKAGE_IMAGE_TAR_NAME: "${{env.PACKAGE_IMAGE_ARTIFACT_NAME_PREFIX}}${{matrix.arch}}.tar"
run: >-
docker load
--input
"${PACKAGE_IMAGE_DIR}/${PACKAGE_IMAGE_TAR_NAME}"

- if: "env.PACKAGE_IMAGE_PUBLISHED == 'true'"
name: "Login to Image Registry"
uses: "docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121" # v4.1.0
with:
registry: "ghcr.io"
username: "${{github.actor}}"
password: "${{secrets.GITHUB_TOKEN}}"

- if: "env.PACKAGE_IMAGE_PUBLISHED == 'true'"
name: "Pull the package image"
working-directory: "${{env.BUILD_DIR}}/clp-package"
shell: "bash"
run: >-
docker pull "$(cat clp-package-image.id)"

- name: "Install uv"
uses: "astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990" # v8.3.2

- name: "Install MariaDB connector"
run: "sudo apt-get update && sudo apt-get install -y libmariadb-dev"

- name: "Run package integration tests"
working-directory: "integration-tests"
run: >-
CLP_BUILD_DIR=../${{env.BUILD_DIR}}
CLP_PACKAGE_DIR=../${{env.BUILD_DIR}}/clp-package
uv run pytest -m 'smoke or package'

- name: "Upload the integration test logs"
if: "always()"
uses: "actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f" # v6.0.0
with:
name: "clp-integration-test-package-logs-${{matrix.arch}}"
path: "${{env.BUILD_DIR}}/integration_tests/test_logs"
retention-days: 3

package-image-multiarch-manifest:
name: "package-image-multiarch-manifest"
if: >-
github.event_name != 'pull_request'
&& github.ref == 'refs/heads/main'
&& needs.package-image.result == 'success'
needs: "package-image"
&& needs.package.result == 'success'
&& needs.package-integration-tests.result == 'success'
needs:
- "package"
- "package-integration-tests"
runs-on: *runner
steps:
- name: "Login to Image Registry"
Expand Down
14 changes: 9 additions & 5 deletions docs/src/dev-docs/tooling-gh-workflows.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ builds, testing, and linting. We briefly describe each workflow below.

This workflow is responsible for:

1. building (Linux) container images containing CLP-core's dependencies,
2. building CLP-core and running its unit tests, and
3. building a container image containing CLP's package components.
1. building (Linux) container images containing CLP-core's dependencies.
2. building CLP-core and running its unit tests.
3. building CLP's package and package image, and running package integration tests.

To minimize build times, the jobs in the workflow are organized in the directed acyclic graph (DAG)
shown below.
Expand Down Expand Up @@ -45,8 +45,9 @@ flowchart LR
musllinux_1_2-deps-image --> musllinux_1_2-x86_64-binaries
musllinux_1_2-deps-image-merge --> musllinux_1_2-x86_64-binaries
ubuntu-jammy-deps-image --> ubuntu-jammy-binaries
ubuntu-jammy-deps-image --> package-image
ubuntu-jammy-deps-image --> package
ubuntu-jammy-binaries --> ubuntu-jammy-binaries-image
package --> package-integration-tests
:::

Arrows between jobs indicate a dependency. The jobs are as follows:
Expand Down Expand Up @@ -75,7 +76,10 @@ Arrows between jobs indicate a dependency. The jobs are as follows:
container and runs core's unit tests.
* `musllinux_1_2-x86_64-binaries`: Builds the CLP-core binaries in the built musllinux_1_2 container
and runs core's unit tests.
* `package-image`: Builds a container image containing CLP's package components.
* `package`: Builds CLP's package, along with a container image containing the package's components,
then uploads the package as a tarball artifact.
* `package-integration-tests`: Runs the package's integration tests against the package built by the
`package` job.
* `ubuntu-jammy-binaries`: Builds the CLP-core binaries in the built Ubuntu Jammy container and runs
core's unit tests.
* `ubuntu-jammy-binaries-image`: Builds an Ubuntu Jammy container image containing CLP-core's
Expand Down
1 change: 1 addition & 0 deletions integration-tests/.pytest.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,5 @@ markers =
core: mark tests that test the CLP core binaries
package: mark tests that use the CLP package
search: mark tests that test search
smoke: mark tests that are smoke tests
startstop: mark tests that test start-stop
2 changes: 2 additions & 0 deletions integration-tests/tests/test_clp_native_py_project_imports.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
"""Smoke tests to validate that CLP Python projects can be imported without errors."""

import pytest
from clp_mcp_server.constants import QueryJobType
from clp_package_utils.general import JobType
from clp_py_utils.clp_config import StorageEngine
from job_orchestration.scheduler.constants import CompressionJobStatus


@pytest.mark.smoke
def test_clp_native_py_project_enum_classes() -> None:
"""
Verifies that the following CLP Python projects can be imported successfully by testing
Expand Down
2 changes: 1 addition & 1 deletion integration-tests/tests/utils/classes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
from tests.conftest import get_test_log_dir
from tests.utils.utils import validate_dir_exists, validate_file_exists

DEFAULT_CMD_TIMEOUT_SECONDS = 120.0
DEFAULT_CMD_TIMEOUT_SECONDS = 300.0

logger = logging.getLogger(__name__)

Expand Down
31 changes: 21 additions & 10 deletions taskfile.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,26 @@ tasks:
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
INCLUDE_PATTERNS: ["{{.OUTPUT_DIR}}"]
cmds:
- task: "package-assemble"
vars:
OUTPUT_DIR: "{{.OUTPUT_DIR}}"
- >-
rsync --archive
"{{.G_BUILD_DIR}}/clp-package-image.id"
"{{.OUTPUT_DIR}}"
- "echo '{{.G_PACKAGE_VERSION}}' > '{{.OUTPUT_DIR}}/VERSION'"
# This command must be last
- task: "utils:checksum:compute"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
INCLUDE_PATTERNS: ["{{.OUTPUT_DIR}}"]

package-assemble:
requires:
vars: ["OUTPUT_DIR"]
deps:
- "generate-config-schemas"
cmds:
- "rm -rf '{{.OUTPUT_DIR}}'"
- >-
Expand All @@ -161,16 +181,6 @@ tasks:
rsync --archive
"tools/deployment/package/"
"{{.OUTPUT_DIR}}"
- >-
rsync --archive
"{{.G_BUILD_DIR}}/clp-package-image.id"
"{{.OUTPUT_DIR}}"
- "echo '{{.G_PACKAGE_VERSION}}' > '{{.OUTPUT_DIR}}/VERSION'"
# This command must be last
- task: "utils:checksum:compute"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
INCLUDE_PATTERNS: ["{{.OUTPUT_DIR}}"]

core:
cmds:
Expand Down Expand Up @@ -246,6 +256,7 @@ tasks:
COMPONENT: "{{.TASK}}"

generate-config-schemas:
run: "once"
vars:
OUTPUT_DIR: "{{.G_CONFIG_SCHEMA_BUILD_DIR}}"
cmds:
Expand Down
Loading