CI Engineering Notes
+A breakdown of the caching strategy in SCIInstitute/shapeworks — what's actually doing the work, what's dead code, and which pattern SCIRun should adopt (it currently caches nothing).
+ShapeWorks has two layers of caching in its GitHub Actions workflows, but only one is wired into the active pipeline. Knowing which is which matters before copying anything.
+ +| Layer | What it caches | Mechanism | Status |
|---|---|---|---|
| Dependency cache | +Compiled 3rd-party C++ libs (ITK, VTK, OpenVDB, etc.) | +GitHub-native actions/cache |
+ Active — the real win | +
| ccache (per-object compiler cache) | +.o files from their own source |
+ scp to a self-hosted SSH host |
+ Legacy / dormant | +
The multi-hour build of ITK/VTK/OpenVDB is cached and reused across every PR and commit. Re-runs only when the dependency build scripts change. This is what keeps their CI fast.
+A self-hosted scp-tarball scheme. Despite the scripts in the repo, no current workflow calls it and CMake is never told to use it. Effectively dead code from the self-hosted-runner era.
Every workflow (build-linux, mac-arm64, mac-intel, windows, linux-debug) uses the same split restore / save pattern:
+ +- name: Restore Dependencies Cache + id: cache-deps-restore + uses: actions/cache/restore@v3 + with: + path: /github/home/install # ~/install on mac, C:\deps on windows + key: ${{ runner.os }}-deps-${{ hashFiles( + '.github/workflows/Dockerfile', + '.github/workflows/gha_deps.sh', + 'install_shapeworks.sh', + 'python_requirements.txt', + 'build_dependencies.sh') }} + +- name: Build Dependencies + if: steps.cache-deps-restore.outputs.cache-hit != 'true' # skip on hit + run: .github/workflows/gha_deps.sh + +- name: Save Dependencies Cache + if: steps.cache-deps-restore.outputs.cache-hit != 'true' # save only on miss + uses: actions/cache/save@v3 + with: + path: ... + key: ... # same key+ +
build_dependencies.sh, gha_deps.sh, install_shapeworks.sh, requirements files, Dockerfile) — not on the project's .cpp/.h files. So the expensive third-party build is reused across every PR and commit, and only re-runs when someone actually changes how dependencies are built.restore + save (not the combined actions/cache@v3), with both the build step and the save step gated on cache-hit != 'true' — the standard "only rebuild on a miss" pattern.-arm64-, -intel-, -deps-debug-, plus osx${MACOSX_DEPLOYMENT_TARGET} and the Linux Docker image hash.gha_deps.sh itself short-circuits with if [ -d ${DEP_PATH} ] before building anything.Takeaway
+Cache the output of the expensive, slow-changing thing (third-party deps), keyed on the files that define how it's built. Everything else recompiles cheaply each run.
+common.sh, restore_caches.sh, and store_ccache.sh implement a self-hosted scheme: scp ccache / conda / dep tarballs (pigz tar.gz, or 7z on Windows) to and from a secret SSH host (secrets.SSH_HOST). common.sh sets USE_CCACHE=ON for mac/linux.
But it's effectively dead code
+Grepping the actual workflow YAMLs, the only ccache reference is brew install ccache in the two Mac jobs. No step calls restore_caches.sh / store_ccache.sh, and CMake is never told to use it (CMAKE_CXX_COMPILER_LAUNCHER=ccache is absent). These scripts are leftovers from an older self-hosted-runner era, superseded by actions/cache.
SCIRun's nine workflows (ccpp.yml, mac.yml, windows.yml, reusable-build.yml, …) use zero caching today — every CI run rebuilds everything from scratch, including the Superbuild externals.
Highest-leverage change
+Copy the ShapeWorks dependency-cache pattern: cache SCIRun's Superbuild / externals output, keyed on the toolchain + Superbuild CMake files, using the same restore → (build if miss) → save shape.
+CMakeLists.txt / external-project files and the toolchain. Invalidates only when the externals definition changes.ccache / sccache as a compiler launcher for SCIRun's own sources — but on GitHub-hosted runners, wire it through actions/cache, not ShapeWorks' scp-to-a-private-host approach.This is especially relevant given SCIRun's C++ modernization work is currently gated on new CI testing jobs landing — faster CI directly unblocks that.
+