feat: add service-catalog pipeline and shared homelab workload test harness

[clubanderson]

Summary

Implements the shared k8s-native service-catalog execution path under #51, closing #79. This unblocks the first media (#80) and non-media (#81) lane implementations.

Argo workflow templates

• service-catalog-pipeline.yaml — top-level pipeline: create namespace, deploy lane manifests, run tests, cleanup on exit. Accepts lane, image-tag, and branch parameters.
• run-service-tests.yaml — non-GNOME test runner: clones testing-lab, discovers tests/service_catalog/<lane>/, runs pytest with JUnit XML output, emits summary to stdout for Argo/Loki.
• service-catalog-smoke.yaml — submit wrapper for convenience.

Shared test helpers (tests/service_catalog/shared/)

• deploy.py — deployment readiness, pod running state, endpoints, event capture
• persistence.py — PVC bound assertion, sentinel file write/read across restart, pod UID change verification
• reachability.py — cluster DNS resolution, HTTP/HTTPS response, TLS handshake
• redeploy.py — image tag upgrade, rollout verification, running image assertion
• teardown.py — namespace existence check, cluster-scoped resource leak detection

Operator entry points

• Justfile — run-service-catalog-smoke with lane/image-tag/branch params
• WORKFLOWS.md — service-catalog-pipeline and run-service-tests documentation
• RUNBOOK.md — architecture diagram, adding-a-lane guide, result inspection, failure modes

How lanes plug in

Each lane provides:

1. tests/service_catalog/<lane>/manifests.yaml — k8s manifests
2. tests/service_catalog/<lane>/test_<lane>.py — pytest suite importing shared helpers

The pipeline is lane-agnostic; it reads the lane name from the lane parameter and discovers manifests + tests by convention.

Links

• Closes feat: add service-catalog pipeline and shared homelab workload test harness #79
• Child of epic: homelab service-catalog workload validation #51
• Unblocks feat: add the first media-service lane to the service-catalog pipeline #80 (media lane) and feat: add the first non-media service lane to the service-catalog pipeline #81 (non-media lane)
• Depends on feat: define service-catalog workload contract for homelab validation #66 (workload contract)

Test plan

• just run-service-catalog-smoke submits the workflow
• Deploy step fails gracefully with clear error when lane manifests are missing
• Test runner fails gracefully when lane test directory is missing
• Namespace cleanup runs on exit regardless of test outcome
• Shared helpers import cleanly and provide the contract-required assertions
• WORKFLOWS.md and RUNBOOK.md accurately describe the new pipeline

[hanthor]

LGTM — well-structured test/infra addition, clean code.

[hanthor]

This is the foundational service-catalog pipeline PR — the remaining service-catalog lane PRs (#193, #194, #195, #196, #197, #198, #199, #200, #201, #203, #204, #205) all depend on this landing first. Merge this one before the lane-specific PRs. LGTM.

[hanthor]

Lane-parameterization verification needed. Foundational PR -- shared helpers must be verified for lane-agnostic operation.

Required verification:

1. Verify all shared helpers (deploy.py, persistence.py, reachability.py, redeploy.py, teardown.py) accept deployment name as parameter, not hardcoded.
2. APP_LABEL default app=service-catalog-workload will not match lane-specific labels like app=homelab-media-service. Tests must override via env vars, and shared modules must accept label selector as parameter.

Builds on: #200 (contract -- merge first)

[castrojo]

Scope check before this stack lands — tagging for maintainer review.

PRs #200–#205 introduce a service-catalog testing layer (Jellyfin, CUPS/OpenPrinting, HTTPS exposure, auth-gating) that runs pytest in-cluster against Kubernetes deployments. This is a different paradigm from the rest of the repo, which provisions fresh KubeVirt VMs and tests Bluefin's atomic OS contract via behave/qecore.

AGENTS.md north star:

"Validate bootc, staged deployments, rollback behavior, read-only /usr, signature policy, composefs/fs-verity, and uupd orchestration before inventing DNF/RPM-style checks."
"Treat Homebrew, Flatpak, Podman, and Docker/Colima as decoupled user-space layers."
"Bias new issues and tests toward immutable-state evidence."

The service-catalog tests prove Kubernetes-native workload contracts (PVC lifecycle, reachability, rollout persistence) rather than Bluefin's atomic OS properties. homelab-substrate, homelab-storage, and homelab-access-probe already exist and seem to be the right base for this kind of in-cluster validation.

Questions for the maintainer:

1. Is the service-catalog direction intentionally expanding scope beyond the atomic OS mandate, or should these tests be framed as "homelab workload contract" tests that are explicitly secondary?
2. The base pipeline PR (feat: add service-catalog pipeline and shared homelab workload test harness #202) introduces service-catalog-pipeline and run-service-tests WorkflowTemplates — ArgoCD will sync these to the cluster. Is that the intent?
3. Should these be gated as optional/experimental so they don't get added to nightlies?

Not blocking — just flagging before 10 PRs land at once. The code itself is well-structured and the test isolation (skip gates, deferred lanes, onExit cleanup) is solid.

[hanthor]

Review: Approved ✅

Part of the structured service-catalog project build-up. Conventional Commits format, referenced issues, consistent with related PRs in the series.