ci: add facility for benchmarking as part of CI (#4745)

Make it so that CI test cases that set the GHA variable "benchmark" to 1
will add a benchmarking step to the workflow that runs a new
ci-benchmark.bash script.

The script runs selected unit tests containing benchmarks (currently,
only image_span_test, but we can amend later as needed). Those
designated tests are run, and their output both echoed to the log for
that step and also put in build/benchmarks/TESTNAME and saved as a build
artifact for optional download.

Most test cases will not turn benchmarking on -- it probably will end up
adding a few minutes so do it very selectively (once per major platform
or compiler version is plenty).

I would have previously guessed that any attempts at benchmarking on GHA
runners was doomed, but in practice, I'm surprised to find that there's
almost as much run-to-run consistency as I find doing casual benchmarks
on my own machine. As such, I think this can be a handy way to do some
rough benchmarking using CI, to compare platforms or compilers, or
verify that changes we want to make don't introduce performance
regressions.

Caveats to remember in the future:

* Take it all with a big grain of salt, and watch the benchmark numbers
for the trial-to-trial range of times -- wide variation means that the
numbers probably can't be trusted.
* The GH runners themselves may change without warning, so beware
benchmark stability over time, or if they ever have pools of
heterogeneous machine generations/configurations.
* While my results indicate a decent amount of timing reliability for
purely computational tests, I assume that there will be enormous
run-to-run variation in anything involving I/O or networking. So this is
unlikely to be a fruitful way of testing for performance regressions in
image format I/O speed (but probably is useful for a variety of
in-memory operations).
* As we add more unit tests to what we benchmark in the future, keep an
eye how much time we're spending running these benchmarks. A few minutes
on a small subset of the test jobs is probably fine, but I wouldn't want
it to make the overall wait for a full CI run to become substantially
longer because of it.

Signed-off-by: Larry Gritz <lg@larrygritz.com>

Author: lgritz

.github/workflows/build-steps.yml

@@ -50,6 +50,8 @@ on:
         type: string
       abi_check:
         type: string
+      benchmark:
+        type: string
       build_docs:
         type: string
       clang_format:
@@ -145,6 +147,10 @@ jobs:
         if: inputs.skip_tests != '1'
         shell: bash
         run: src/build-scripts/ci-test.bash
+      - name: Benchmarks
+        if: inputs.benchmark == '1'
+        shell: bash
+        run: src/build-scripts/ci-benchmark.bash
       - name: clang-format
         if: inputs.clang_format == '1'
         shell: bash
@@ -193,13 +199,14 @@ jobs:
             time make sphinx
       - name: Upload testsuite debugging artifacts
         uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
-        if: ${{ failure() || inputs.build_docs == '1'}}
+        if: ${{ failure() || inputs.build_docs == '1' || inputs.benchmark == '1' }}
         with:
           name: oiio-${{github.job}}-${{inputs.nametag}}
           path: |
             build/cmake-save
             build/compat_reports
             build/sphinx
+            build/benchmarks
             build/testsuite/*/*.*
             !build/testsuite/oiio-images
             !build/testsuite/openexr-images

.github/workflows/ci.yml

@@ -200,6 +200,10 @@ jobs:
       - name: Testsuite
         if: matrix.skip_tests != '1'
         run: src/build-scripts/ci-test.bash
+      - name: Benchmarks
+        if: matrix.benchmark == '1'
+        shell: bash
+        run: src/build-scripts/ci-benchmark.bash
       - name: Check out ABI standard
         if: matrix.abi_check != ''
         uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
@@ -232,6 +236,7 @@ jobs:
             build/cmake-save
             build/compat_reports
             build/sphinx
+            build/benchmarks
             build/testsuite/*/*.*
             !build/testsuite/oiio-images
             !build/testsuite/openexr-images
@@ -267,6 +272,7 @@ jobs:
       skip_build: ${{ matrix.skip_build }}
       skip_tests: ${{ matrix.skip_tests }}
       abi_check: ${{ matrix.abi_check }}
+      benchmark: ${{ matrix.benchmark }}
       build_docs: ${{ matrix.build_docs }}
       clang_format: ${{ matrix.clang_format }}
       generator: ${{ matrix.generator }}
@@ -312,6 +318,7 @@ jobs:
             python_ver: "3.10"
             pybind11_ver: v2.10.0
             simd: "avx2,f16c"
+            benchmark: 1
             setenvs: export USE_OPENVDB=0
                             xOPENCOLORIO_CXX=g++
                             UHDR_CMAKE_C_COMPILER=gcc
@@ -326,6 +333,7 @@ jobs:
             simd: "avx2,f16c"
             fmt_ver: 10.1.1
             pybind11_ver: v2.12.0
+            benchmark: 1
             setenvs: PUGIXML_VERSION=v1.14
           - desc: VFX2024 clang/C++17 py3.11 exr3.2 ocio2.3
             nametag: linux-vfx2024.clang
@@ -337,6 +345,7 @@ jobs:
             simd: "avx2,f16c"
             fmt_ver: 10.1.1
             pybind11_ver: v2.12.0
+            benchmark: 1
             setenvs: PUGIXML_VERSION=v1.14
           - desc: VFX2025 gcc11/C++17 py3.11 exr3.3 ocio2.4
             nametag: linux-vfx2025
@@ -346,6 +355,7 @@ jobs:
             simd: "avx2,f16c"
             fmt_ver: 11.1.4
             pybind11_ver: v2.13.6
+            benchmark: 1
             setenvs: PUGIXML_VERSION=v1.15
           - desc: Sanitizers
             nametag: sanitizer
@@ -431,6 +441,7 @@ jobs:
             pybind11_ver: master
             python_ver: "3.12"
             simd: avx2,f16c
+            benchmark: 1
             setenvs: export LIBJPEGTURBO_VERSION=main
                             LIBRAW_VERSION=master
                             LIBTIFF_VERSION=master
@@ -529,6 +540,7 @@ jobs:
       simd: ${{ matrix.simd }}
       skip_build: ${{ matrix.skip_build }}
       skip_tests: ${{ matrix.skip_tests }}
+      benchmark: ${{ matrix.benchmark }}
       abi_check: ${{ matrix.abi_check }}
       build_docs: ${{ matrix.build_docs }}
       generator: ${{ matrix.generator }}
@@ -550,6 +562,7 @@ jobs:
             simd: sse4.2,avx2
             ctest_test_timeout: 1200
             setenvs: export MACOSX_DEPLOYMENT_TARGET=12.0
+            benchmark: 1
           - desc: MacOS-14-ARM aclang15/C++20/py3.12
             runner: macos-14
             nametag: macos14-arm-py312
@@ -564,6 +577,7 @@ jobs:
             cxx_compiler: clang++
             cxx_std: 20
             python_ver: "3.13"
+            benchmark: 1
 
 
   #
@@ -592,6 +606,7 @@ jobs:
       simd: ${{ matrix.simd }}
       skip_build: ${{ matrix.skip_build }}
       skip_tests: ${{ matrix.skip_tests }}
+      benchmark: ${{ matrix.benchmark }}
       abi_check: ${{ matrix.abi_check }}
       build_docs: ${{ matrix.build_docs }}
       generator: ${{ matrix.generator }}
@@ -624,3 +639,4 @@ jobs:
             generator: "Visual Studio 17 2022"
             python_ver: "3.9"
             setenvs: export OPENIMAGEIO_PYTHON_LOAD_DLLS_FROM_PATH=1
+            benchmark: 1

src/build-scripts/ci-benchmark.bash

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+
+# Copyright Contributors to the OpenImageIO project.
+# SPDX-License-Identifier: Apache-2.0
+# https://github.com/AcademySoftwareFoundation/OpenImageIO
+
+BUILD_BIN_DIR=build/bin
+if [[ "${RUNNER_OS}" == "Windows" ]] ; then
+    BUILD_BIN_DIR+=/${CMAKE_BUILD_TYPE}
+fi
+
+ls build
+ls $BUILD_BIN_DIR
+
+mkdir -p build/benchmarks
+for t in image_span_test ; do
+    echo
+    echo
+    echo "$t"
+    echo "========================================================"
+    ${BUILD_BIN_DIR}/$t > build/benchmarks/$t.out
+    cat build/benchmarks/$t.out
+    echo "========================================================"
+    echo "========================================================"
+    echo
+done