feat: add service-catalog pipeline and shared homelab workload test harness

Justfile

@@ -292,6 +292,19 @@ run-homelab-restore:
     argo submit --from workflowtemplate/homelab-restore-drill \
       -n {{ argo_ns }} --wait --log
 
+# ── Service-catalog workload validation (#51) ────────────────────────────────
+
+# Run service-catalog pipeline for a given lane (default: media)
+# Usage: just run-service-catalog-smoke
+# Usage: just run-service-catalog-smoke lane=non-media
+# Usage: just run-service-catalog-smoke lane=media image-tag=lts branch=feat/my-branch
+run-service-catalog-smoke lane="media" image-tag="latest" branch="main":
+    argo submit --from workflowtemplate/service-catalog-pipeline \
+      -p lane={{ lane }} \
+      -p image-tag={{ image-tag }} \
+      -p branch={{ branch }} \
+      -n {{ argo_ns }} --wait --log
+
 # ── Ghost maintenance ─────────────────────────────────────────────────────────
 
 # Patch ghost OTel collector config to remove noisy process scraper (#117)

RUNBOOK.md

@@ -84,6 +84,45 @@ Golden disks can be patched by workflow after key rotation; titan disk key refre
 - `findChild(..., requireResult=...)` is not a supported dogtail pattern in this repo's stack.
 - `findChildren(...)` and `findChild(..., retry=False)` are the canonical presence-check APIs.
 
+## Service-catalog pipeline
+
+A second execution path validates homelab service workloads directly in
+Kubernetes, without VMs or GNOME infrastructure:
+
+```
+Argo Workflow (argo namespace)
+        │
+        ├─ create-namespace       ─► ephemeral namespace svc-<lane>-<uid>
+        ├─ deploy-workload        ─► clone repo, kubectl apply lane manifests
+        ├─ run-service-tests      ─► pytest against live k8s workload
+        └─ cleanup (onExit)       ─► delete namespace
+```
+
+| Property | Value |
+|---|---|
+| Entry point | `just run-service-catalog-smoke` or `argo submit --from workflowtemplate/service-catalog-pipeline` |
+| Parameters | `lane` (default: media), `image-tag` (default: latest), `branch` (default: main) |
+| Wall-clock | ~3–5 min |
+| Evidence | pytest JUnit XML + per-check artifacts in pod logs (Loki) |
+
+### Adding a new lane
+
+1. Create `tests/service_catalog/<lane>/manifests.yaml` with the workload's
+   Kubernetes manifests (Deployment, Service, PVC, Secrets, ConfigMaps).
+2. Create `tests/service_catalog/<lane>/test_<lane>.py` importing helpers
+   from `tests/service_catalog/shared/` (deploy, persistence, reachability,
+   redeploy, teardown).
+3. Run: `just run-service-catalog-smoke lane=<lane>`
+4. The pipeline creates a namespace, applies manifests, runs tests, cleans up.
+
+### Inspecting results
+
+- `argo logs @latest` — test output including summary and artifact list.
+- Loki query: `{namespace="argo"} |= "svc-catalog"` — all service-catalog
+  workflow logs.
+- JUnit XML is emitted to the pod stdout and captured by Argo's log
+  archival. No separate artifact upload path is needed.
+
 ## Common failure modes
 
 | Symptom | Root cause | Durable fix |
@@ -96,6 +135,9 @@ Golden disks can be patched by workflow after key rotation; titan disk key refre
 | VM stuck `Terminating` | KubeVirt controller race with launcher cleanup | Delete the `virt-launcher-*` pod and let reconciliation finish |
 | `run-gnome-tests` pod fails at startup | Workflow template structure error, often misplaced `volumes:` | Fix the template in git and let ArgoCD reconcile it |
 | WorkflowTemplate change appears ignored | Workflow was submitted before the new template was reconciled | Verify ArgoCD revision, wait or sync, then submit a new workflow |
+| Service-catalog deploy step fails with "No manifests found" | Lane directory missing `manifests.yaml` | Create `tests/service_catalog/<lane>/manifests.yaml` per the contract |
+| Service-catalog test step fails with "No test suite" | Lane directory missing under `tests/service_catalog/` | Create the lane test directory with at least one `test_*.py` file |
+| Service-catalog namespace stuck terminating | Finalizer or PVC not released | Check for stuck PVCs or pods with `kubectl get all -n <ns>`, delete manually if needed |
 
 ## Historical notes
 

WORKFLOWS.md

@@ -99,13 +99,59 @@ selinux=0, sudoers) on an existing golden disk without rebuilding it.
 |---|---|---|
 | `image-tag` | `latest` | Disk dir under `/var/tmp/bluefin-golden/`. |
 
+### `service-catalog-pipeline`
+
+K8s-native service-catalog validation pipeline. Deploys a lane's workload
+manifests into an ephemeral namespace, runs the lane's pytest suite, and
+tears down on exit. Does not use VMs or GNOME infrastructure — runs
+directly against k8s-hosted workloads.
+
+| Parameter | Default | Notes |
+|---|---|---|
+| `lane` | `media` | Lane name — must match a directory under `tests/service_catalog/<lane>/` and `tests/service_catalog/<lane>/manifests.yaml` |
+| `image-tag` | `latest` | Passed to lane manifests (available for future per-lane image selection) |
+| `branch` | `main` | testing-lab branch to clone for manifests and tests |
+
+Wall-clock: ~3–5 min (depends on image pull and test count).
+
+```
+just run-service-catalog-smoke                        # media lane, latest
+just run-service-catalog-smoke lane=non-media         # non-media lane
+just run-service-catalog-smoke lane=media branch=feat/my-branch
+```
+
+Pipeline structure:
+```
+create-namespace → deploy-workload → run-tests → cleanup (onExit)
+```
+
+The deploy step reads `tests/service_catalog/<lane>/manifests.yaml` from
+the cloned repo and applies it to the ephemeral namespace. Each lane owns
+its own manifests — the pipeline is lane-agnostic.
+
+Test runner: `run-service-tests` (see supporting templates below). Uses
+the shared helpers in `tests/service_catalog/shared/` for deployment,
+persistence, reachability, redeploy, and teardown assertions.
+
 ---
 
 ## Supporting templates (called via `templateRef`)
 
 These are exposed only because they are referenced by the entry points;
 submit them directly only for diagnosis.
 
+### `run-service-tests` (template: `run-pytest`)
+
+Non-GNOME test runner for service-catalog lanes. Clones testing-lab,
+discovers the test suite under `tests/service_catalog/<lane>/`, and runs
+pytest with JUnit XML output. Emits a summary line (`N/M pytest checks
+passed`) to stdout for Argo/Loki consumption.
+
+Env vars passed to the test container: `TEST_NAMESPACE`, `TEST_LANE`,
+`TEST_RESULTS_DIR`. Lane-specific tests import shared helpers from
+`tests/service_catalog/shared/` (deploy, persistence, reachability,
+redeploy, teardown).
+
 ### `bib-build-and-push` (template: `ensure-disk`)
 
 Builds the golden raw disk via `bootc-image-builder` if missing or stale.

argo/service-catalog-smoke.yaml

@@ -0,0 +1,17 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Workflow
+metadata:
+  generateName: svc-catalog-
+  namespace: argo
+spec:
+  serviceAccountName: argo
+  workflowTemplateRef:
+    name: service-catalog-pipeline
+  arguments:
+    parameters:
+      - name: lane
+        value: "media"
+      - name: image-tag
+        value: "latest"
+      - name: branch
+        value: "main"

argo/workflow-templates/run-service-tests.yaml

@@ -0,0 +1,126 @@
+apiVersion: argoproj.io/v1alpha1
+kind: WorkflowTemplate
+metadata:
+  name: run-service-tests
+  namespace: argo
+  annotations:
+    bluefin.io/stack: "k8s workload + pytest"
+spec:
+  templates:
+    - name: run-pytest
+      metadata:
+        labels:
+          app.kubernetes.io/part-of: bluefin-test-suite
+          bluefin.io/step: run-service-tests
+          bluefin.io/lane: "svc-{{inputs.parameters.lane}}"
+      inputs:
+        parameters:
+          - name: namespace
+          - name: lane
+          - name: branch
+            value: "main"
+      activeDeadlineSeconds: 1800
+      initContainers:
+        - name: git-sync
+          image: quay.io/fedora/fedora:latest
+          env:
+            - name: BRANCH
+              value: "{{inputs.parameters.branch}}"
+          command: [bash, -c]
+          args:
+            - |
+              set -euo pipefail
+              dnf install -y --quiet git 2>&1 | tail -2
+              for attempt in 1 2 3; do
+                rm -rf /workspace/testing-lab
+                if git clone --depth 1 --branch "${BRANCH}" \
+                    https://github.com/projectbluefin/testing-lab \
+                    /workspace/testing-lab 2>&1; then
+                  echo "Synced testing-lab @ ${BRANCH} (attempt ${attempt})"
+                  exit 0
+                fi
+                rm -rf /workspace/testing-lab
+                if git clone https://github.com/projectbluefin/testing-lab /workspace/testing-lab 2>&1 && \
+                   git -C /workspace/testing-lab checkout "${BRANCH}" 2>&1; then
+                  echo "Synced testing-lab @ ${BRANCH} via SHA fallback (attempt ${attempt})"
+                  exit 0
+                fi
+                echo "Attempt ${attempt} failed, retrying in 15s..."
+                sleep 15
+              done
+              exit 1
+          volumeMounts:
+            - name: workspace
+              mountPath: /workspace
+      container:
+        image: quay.io/fedora/fedora:latest
+        imagePullPolicy: Always
+        resources:
+          requests:
+            cpu: "500m"
+            memory: 512Mi
+          limits:
+            cpu: "2"
+            memory: 2Gi
+        env:
+          - name: TEST_NAMESPACE
+            value: "{{inputs.parameters.namespace}}"
+          - name: TEST_LANE
+            value: "{{inputs.parameters.lane}}"
+          - name: TEST_RESULTS_DIR
+            value: /tmp/results
+        volumeMounts:
+          - name: workspace
+            mountPath: /workspace
+            readOnly: true
+        command: [bash, -c]
+        args:
+          - |
+            set -euo pipefail
+            mkdir -p /tmp/results
+            LOG=/tmp/results/runner.log
+            exec > >(tee -a "${LOG}") 2>&1
+
+            dnf install -y --quiet python3 python3-pytest kubernetes-client jq curl openssl 2>&1 | tail -4
+
+            LANE="{{inputs.parameters.lane}}"
+            SUITE="/workspace/testing-lab/tests/service_catalog/${LANE}"
+
+            if [[ ! -d "${SUITE}" ]]; then
+              echo "ERROR: No test suite at ${SUITE}" >&2
+              exit 1
+            fi
+
+            export PYTHONPATH="/workspace/testing-lab:${PYTHONPATH:-}"
+
+            TEST_RC=0
+            python3 -m pytest -q "${SUITE}" \
+              --junitxml /tmp/results/pytest-results.xml || TEST_RC=$?
+
+            echo "Collected result files:" >&2
+            find /tmp/results -maxdepth 1 -type f -printf '%f\n' | sort >&2
+
+            if [[ -f /tmp/results/pytest-results.xml ]]; then
+              SUMMARY="$(python3 -c "
+            import xml.etree.ElementTree as ET
+            root = ET.parse('/tmp/results/pytest-results.xml').getroot()
+            suites = [root] if root.tag == 'testsuite' else root.findall('testsuite')
+            tests = sum(int(s.attrib.get('tests', 0)) for s in suites)
+            skipped = sum(int(s.attrib.get('skipped', 0)) for s in suites)
+            failures = sum(int(s.attrib.get('failures', 0)) + int(s.attrib.get('errors', 0)) for s in suites)
+            passed = tests - skipped - failures
+            print(f'{passed}/{tests} pytest checks passed')
+            " 2>/dev/null || true)"
+              if [[ -n "${SUMMARY}" ]]; then
+                printf '%s\n' "${SUMMARY}"
+              else
+                echo "pytest results present"
+              fi
+            else
+              echo "pytest results missing"
+            fi
+
+            exit "${TEST_RC}"
+      volumes:
+        - name: workspace
+          emptyDir: {}