diff --git a/Justfile b/Justfile index 37cb4e689..c95b72b36 100644 --- a/Justfile +++ b/Justfile @@ -292,6 +292,13 @@ run-homelab-restore: argo submit --from workflowtemplate/homelab-restore-drill \ -n {{ argo_ns }} --wait --log +# ── Service-catalog workload lanes ─────────────────────────────────────────── + +# Run the non-media (OpenPrinting/CUPS base) service-catalog lane (#81) +run-service-nonmedia: + argo submit --from workflowtemplate/homelab-nonmedia-service \ + -n {{ argo_ns }} --wait --log + # ── Ghost maintenance ───────────────────────────────────────────────────────── # Patch ghost OTel collector config to remove noisy process scraper (#117) diff --git a/RUNBOOK.md b/RUNBOOK.md index 4f4f34a85..ee3734a51 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -96,6 +96,10 @@ Golden disks can be patched by workflow after key rotation; titan disk key refre | VM stuck `Terminating` | KubeVirt controller race with launcher cleanup | Delete the `virt-launcher-*` pod and let reconciliation finish | | `run-gnome-tests` pod fails at startup | Workflow template structure error, often misplaced `volumes:` | Fix the template in git and let ArgoCD reconcile it | | WorkflowTemplate change appears ignored | Workflow was submitted before the new template was reconciled | Verify ArgoCD revision, wait or sync, then submit a new workflow | +| `PVC nonmedia-service-config not Bound` in service-catalog lane | No default StorageClass or provisioner not ready | Verify `kubectl get storageclass`; ensure the local-path provisioner pod is Running on ghost | +| `nonmedia-service` pod stuck `Pending` in lane tests | nodeSelector `ghost` doesn't match any Ready node | Check `kubectl get nodes` — exo-1 is ineligible; ghost must be Ready | +| Lane test suite skipped entirely | `TEST_NAMESPACE` env var unset or empty | Confirm the runner pod is launched from the `run-incluster-tests` template, not directly | +| Cleanup step hangs waiting for namespace deletion | Finalizers on PVCs blocking namespace GC | Run `kubectl get pvc -n ` and remove stuck finalizers with `kubectl patch pvc … -p '{"metadata":{"finalizers":[]}}'` | ## Historical notes diff --git a/WORKFLOWS.md b/WORKFLOWS.md index a0074da42..4cda02aa2 100644 --- a/WORKFLOWS.md +++ b/WORKFLOWS.md @@ -170,6 +170,55 @@ just run-dakota-build all # both variants sequentially --- +## Service-catalog workload lanes + +In-cluster validation suites that deploy a fixture workload into a per-run namespace, +run pytest checks against the live Kubernetes objects, and clean up on exit. +Each lane is a self-contained `WorkflowTemplate` with the DAG: +`create-namespace → deploy-fixture → run-tests (onExit: cleanup)`. + +All lanes delegate the test-runner pod to the shared `run-incluster-tests / run-pytest` +template, which clones the testing-lab repo and runs `pytest -q `. + +### `homelab-nonmedia-service` (issue #81) + +Validates the base homelab workload contract for network-only print services +(OpenPrinting/CUPS class, bluespeed#11). The fixture is a lightweight nginx stub +on port 631 (IPP-standard) so the contract can be proven without real CUPS hardware. + +| Parameter | Default | Notes | +|---|---|---| +| `lane` | `nonmedia` | Fixed; used as `TEST_LANE` env var inside the runner pod. | +| `branch` | `main` | testing-lab git ref cloned by the runner. | + +Suite path: `tests/service_catalog/nonmedia/test_nonmedia_lane.py` (15 tests; 2 explicit skips for USB and mDNS hardware — see issue #67). + +``` +just run-service-nonmedia # submit + stream logs, ~5–8 min +``` + +Out-of-scope (separate lanes): +- USB printer device hostPath passthrough → `homelab-print-device` (issue #67) +- avahi mDNS / LAN discovery → `homelab-print-device` (issue #67) +- GPU transcoding → `homelab-media-gpu` (issue #63) + +### `bluefin-service-catalog-pipeline` (issue #81) + +Top-level aggregation pipeline for the service-catalog suite. Routes to the +matching lane WorkflowTemplate based on the `lane` parameter. + +| Parameter | Default | Valid values | +|---|---|---| +| `lane` | `nonmedia` | `nonmedia` (first lane; extend as new lanes are added) | +| `branch` | `main` | testing-lab git ref | + +``` +argo submit --from workflowtemplate/bluefin-service-catalog-pipeline \ + -n argo -p lane=nonmedia --wait --log +``` + +--- + ## CronWorkflows Lives in `manifests/`, applied via the `testing-lab-infra` ArgoCD app: diff --git a/argo/homelab-nonmedia-service.yaml b/argo/homelab-nonmedia-service.yaml new file mode 100644 index 000000000..e8d86b040 --- /dev/null +++ b/argo/homelab-nonmedia-service.yaml @@ -0,0 +1,15 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Workflow +metadata: + generateName: nonmedia-service- + namespace: argo +spec: + serviceAccountName: argo + workflowTemplateRef: + name: homelab-nonmedia-service + arguments: + parameters: + - name: lane + value: "nonmedia" + - name: branch + value: "main" diff --git a/argo/workflow-templates/bluefin-service-catalog-pipeline.yaml b/argo/workflow-templates/bluefin-service-catalog-pipeline.yaml new file mode 100644 index 000000000..ce4387a4f --- /dev/null +++ b/argo/workflow-templates/bluefin-service-catalog-pipeline.yaml @@ -0,0 +1,52 @@ +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: bluefin-service-catalog-pipeline + namespace: argo + labels: + app.kubernetes.io/component: service-catalog-pipeline + app.kubernetes.io/part-of: bluefin-test-suite +spec: + entrypoint: pipeline + serviceAccountName: argo + # ── parameters ────────────────────────────────────────────────────────────── + arguments: + parameters: + # Lane selector: controls which service-catalog workload is exercised. + # Valid values: nonmedia (media lanes are in separate templates) + - name: lane + value: "nonmedia" + # git ref the in-cluster runner clones when executing pytest suites. + - name: branch + value: "main" + + # ── pipeline DAG ──────────────────────────────────────────────────────────── + templates: + - name: pipeline + dag: + tasks: + # Route execution to the matching lane WorkflowTemplate. + # New lanes are added as additional when/templateRef entries here. + - name: nonmedia-lane + template: run-nonmedia-lane + when: "\"{{workflow.parameters.lane}}\" == \"nonmedia\"" + arguments: + parameters: + - name: branch + value: "{{workflow.parameters.branch}}" + + - name: run-nonmedia-lane + inputs: + parameters: + - name: branch + steps: + - - name: execute + templateRef: + name: homelab-nonmedia-service + template: pipeline + arguments: + parameters: + - name: lane + value: "nonmedia" + - name: branch + value: "{{inputs.parameters.branch}}" diff --git a/argo/workflow-templates/homelab-nonmedia-service.yaml b/argo/workflow-templates/homelab-nonmedia-service.yaml new file mode 100644 index 000000000..6b6be2577 --- /dev/null +++ b/argo/workflow-templates/homelab-nonmedia-service.yaml @@ -0,0 +1,182 @@ +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: homelab-nonmedia-service + namespace: argo + labels: + app.kubernetes.io/component: homelab-nonmedia-service + app.kubernetes.io/part-of: bluefin-test-suite +spec: + entrypoint: pipeline + onExit: cleanup + serviceAccountName: homelab-runner + arguments: + parameters: + - name: lane + value: "nonmedia" + - name: branch + value: "main" + templates: + - name: pipeline + dag: + tasks: + - name: create-namespace + template: create-namespace + arguments: + parameters: + - name: namespace + value: "nonmedia-service-{{workflow.uid}}" + - name: deploy-fixture + depends: "create-namespace.Succeeded" + template: deploy-fixture + arguments: + parameters: + - name: namespace + value: "nonmedia-service-{{workflow.uid}}" + - name: lane + value: "{{workflow.parameters.lane}}" + - name: run-tests + depends: "deploy-fixture.Succeeded" + templateRef: + name: run-incluster-tests + template: run-pytest + arguments: + parameters: + - name: namespace + value: "nonmedia-service-{{workflow.uid}}" + - name: suite-path + value: "service_catalog/nonmedia" + - name: lane + value: "{{workflow.parameters.lane}}" + - name: app-label + value: "app=nonmedia-service" + - name: service-name + value: "nonmedia-service" + - name: branch + value: "{{workflow.parameters.branch}}" + + - name: create-namespace + inputs: + parameters: + - name: namespace + resource: + action: create + manifest: | + apiVersion: v1 + kind: Namespace + metadata: + name: "{{inputs.parameters.namespace}}" + labels: + app.kubernetes.io/part-of: bluefin-test-suite + bluefin.io/lane: nonmedia + + - name: deploy-fixture + inputs: + parameters: + - name: namespace + - name: lane + script: + image: cgr.dev/chainguard/kubectl:latest-dev + command: [bash, -c] + source: | + set -euo pipefail + NS="{{inputs.parameters.namespace}}" + + # PVC must be applied before the Deployment so the volume mount + # is satisfiable when the first pod is scheduled. + kubectl apply -n "${NS}" -f - <<'EOF' + apiVersion: v1 + kind: PersistentVolumeClaim + metadata: + name: nonmedia-service-config + labels: + app: nonmedia-service + app.kubernetes.io/part-of: bluefin-test-suite + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + EOF + + kubectl apply -n "${NS}" -f - <&2 + + - name: cleanup + script: + image: cgr.dev/chainguard/kubectl:latest-dev + command: [bash, -c] + source: | + set -euo pipefail + NS="nonmedia-service-{{workflow.uid}}" + kubectl delete namespace "${NS}" --ignore-not-found=true >&2 || true + timeout 180 bash -c "until ! kubectl get namespace \"${NS}\" >/dev/null 2>&1; do sleep 5; done" >&2 || true + echo "cleanup-complete" diff --git a/tests/service_catalog/nonmedia/__init__.py b/tests/service_catalog/nonmedia/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/service_catalog/nonmedia/fixtures/nonmedia-pvc.yaml b/tests/service_catalog/nonmedia/fixtures/nonmedia-pvc.yaml new file mode 100644 index 000000000..4fdf7a6cd --- /dev/null +++ b/tests/service_catalog/nonmedia/fixtures/nonmedia-pvc.yaml @@ -0,0 +1,16 @@ +--- +# PVC fixture for the non-media service lane (config volume). +# Applied first so the Deployment can reference it immediately. +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: nonmedia-service-config + labels: + app: nonmedia-service + app.kubernetes.io/part-of: bluefin-test-suite +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi diff --git a/tests/service_catalog/nonmedia/fixtures/nonmedia-service.yaml b/tests/service_catalog/nonmedia/fixtures/nonmedia-service.yaml new file mode 100644 index 000000000..101965eb8 --- /dev/null +++ b/tests/service_catalog/nonmedia/fixtures/nonmedia-service.yaml @@ -0,0 +1,84 @@ +--- +# Fixture: OpenPrinting/CUPS non-media service lane +# Namespace label and PVC are created by the workflow; this manifest +# creates the Deployment and Service that the lane tests validate. +# +# Parameterised fields (substituted by the deploy-fixture template step): +# NAMESPACE — per-run namespace injected by the workflow +# LANE — workflow parameter, default "nonmedia" +# +# Out-of-scope for this base fixture: +# - USB printer device hostPath → issue #67 +# - avahi sidecar / NodePort LB → issue #67 +# - GPU or hardware passthrough → not applicable +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nonmedia-service + labels: + app: nonmedia-service + app.kubernetes.io/part-of: bluefin-test-suite + bluefin.io/workload-class: nonmedia-service +spec: + replicas: 1 + selector: + matchLabels: + app: nonmedia-service + template: + metadata: + labels: + app: nonmedia-service + app.kubernetes.io/part-of: bluefin-test-suite + bluefin.io/workload-class: nonmedia-service + spec: + nodeSelector: + kubernetes.io/hostname: ghost + containers: + - name: print-server + # Representative linuxserver.io-style fixture for the + # OpenPrinting/CUPS workload class (bluespeed#11). + # Uses nginx as a lightweight proxy for the HTTP-reachability + # contract on port 631 (IPP standard). Replace with + # lscr.io/linuxserver/cups when the lane graduates from + # validation to integration testing. + image: docker.io/library/nginx:1.27.5-alpine + ports: + - containerPort: 631 + env: + - name: PUID + value: "1000" + - name: PGID + value: "1000" + - name: TZ + value: "UTC" + volumeMounts: + - name: config + mountPath: /config + # nginx listens on 80 by default; rewrite the listen port to + # 631 so the reachability contract matches the IPP standard + # without requiring a custom image. + command: + - sh + - -c + - | + sed -i 's/listen\s*80;/listen 631;/g' /etc/nginx/conf.d/default.conf + exec nginx -g 'daemon off;' + volumes: + - name: config + persistentVolumeClaim: + claimName: nonmedia-service-config +--- +apiVersion: v1 +kind: Service +metadata: + name: nonmedia-service + labels: + app: nonmedia-service + app.kubernetes.io/part-of: bluefin-test-suite +spec: + selector: + app: nonmedia-service + ports: + - name: ipp + port: 631 + targetPort: 631 diff --git a/tests/service_catalog/nonmedia/test_nonmedia_lane.py b/tests/service_catalog/nonmedia/test_nonmedia_lane.py new file mode 100644 index 000000000..42e68e5bc --- /dev/null +++ b/tests/service_catalog/nonmedia/test_nonmedia_lane.py @@ -0,0 +1,204 @@ +""" +Non-media (OpenPrinting/CUPS) service-catalog lane test suite. + +Validates the base homelab workload contract for network-only print services: + 1. Workload deploys and becomes ready within timeout. + 2. Config PVC is present and Bound. + 3. IPP port 631 is reachable at the service address. + 4. Required environment variables are injected. + 5. A rollout restart produces a fresh Ready pod that remains reachable. + 6. Observability: lane/env metadata is available in pod labels. + 7. Namespace and fixture are cleaned up by the workflow teardown step. + +Out-of-scope for this lane (see issues #67 and #63): + - USB printer device hostPath passthrough + - avahi mDNS / LAN discovery + - GPU or hardware transcoding +""" + +from __future__ import annotations + +import os +import urllib.request + +import pytest + +from tests.service_catalog.shared.kube import ( + NAMESPACE, + APP_LABEL, + SERVICE_NAME, + TEST_LANE, + RESULTS_DIR, + run_kubectl, + require_kubectl, + write_artifact, + get_pods_json, + first_pod_name, +) + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +_IPP_PORT = int(os.environ.get("TEST_IPP_PORT", "631")) +_ROLLOUT_TIMEOUT = "300s" + + +def _http_get_ipp() -> str: + url = f"http://{SERVICE_NAME}.{NAMESPACE}.svc.cluster.local:{_IPP_PORT}/" + with urllib.request.urlopen(url, timeout=20) as response: # noqa: S310 + body = response.read().decode() + write_artifact(f"{TEST_LANE}-ipp-body.txt", body) + return body + + +def _rollout_restart() -> None: + restart = run_kubectl( + "rollout", "restart", f"deployment/{SERVICE_NAME}", "-n", NAMESPACE + ) + assert restart.returncode == 0, restart.stdout + restart.stderr + write_artifact(f"{TEST_LANE}-restart.txt", restart.stdout + restart.stderr) + status = run_kubectl( + "rollout", + "status", + f"deployment/{SERVICE_NAME}", + "-n", + NAMESPACE, + f"--timeout={_ROLLOUT_TIMEOUT}", + ) + assert status.returncode == 0, status.stdout + status.stderr + write_artifact(f"{TEST_LANE}-rollout-status.txt", status.stdout + status.stderr) + + +# --------------------------------------------------------------------------- +# Test cases +# --------------------------------------------------------------------------- + + +class TestNonmediaLaneDeployment: + """Validate the base workload contract for network-only print services.""" + + def test_namespace_exists(self): + """Namespace created by the workflow's create-namespace step must exist.""" + out = require_kubectl("get", "namespace", NAMESPACE, "-o", "name") + assert NAMESPACE in out + + def test_deployment_exists(self): + """Deployment fixture applied by deploy-fixture step must be present.""" + out = require_kubectl( + "get", "deployment", SERVICE_NAME, "-n", NAMESPACE, "-o", "name" + ) + assert SERVICE_NAME in out + + def test_deployment_ready(self): + """All desired replicas must reach Ready state before the test window.""" + out = require_kubectl( + "rollout", + "status", + f"deployment/{SERVICE_NAME}", + "-n", + NAMESPACE, + "--timeout=300s", + ) + write_artifact(f"{TEST_LANE}-deploy-status.txt", out) + assert "successfully rolled out" in out + + def test_pod_running(self): + """At least one pod must be in Running phase.""" + data = get_pods_json() + items = data.get("items", []) + assert items, "No pods found for the nonmedia-service workload" + phases = {p["status"].get("phase") for p in items} + assert "Running" in phases, f"No Running pod; phases observed: {phases}" + + def test_pod_labels_contain_app(self): + """Pod must carry the app= label used by the lane selector.""" + pod_name = first_pod_name() + out = require_kubectl( + "get", "pod", pod_name, "-n", NAMESPACE, "-o", + "jsonpath={.metadata.labels.app}" + ) + assert out.strip() == SERVICE_NAME, ( + f"Expected label app={SERVICE_NAME}, got: {out.strip()!r}" + ) + + def test_pvc_bound(self): + """Config PVC must be Bound before the fixture is considered healthy.""" + pvc_name = f"{SERVICE_NAME}-config" + out = require_kubectl( + "get", "pvc", pvc_name, "-n", NAMESPACE, "-o", + "jsonpath={.status.phase}" + ) + write_artifact(f"{TEST_LANE}-pvc-phase.txt", out) + assert out.strip() == "Bound", f"PVC {pvc_name} is not Bound: {out.strip()!r}" + + def test_service_exists(self): + """A Kubernetes Service exposing port 631 must exist in the namespace.""" + out = require_kubectl( + "get", "service", SERVICE_NAME, "-n", NAMESPACE, "-o", "name" + ) + assert SERVICE_NAME in out + + def test_service_port_631(self): + """Service spec must expose the IPP-standard port 631.""" + out = require_kubectl( + "get", "service", SERVICE_NAME, "-n", NAMESPACE, "-o", + "jsonpath={.spec.ports[*].port}" + ) + ports = [int(p) for p in out.split()] + assert _IPP_PORT in ports, ( + f"Expected port {_IPP_PORT} in service spec; found: {ports}" + ) + + def test_ipp_port_reachable(self): + """HTTP GET on port 631 at the cluster-internal service address must succeed.""" + body = _http_get_ipp() + assert body is not None and len(body) >= 0 # any HTTP 2xx is sufficient + + def test_env_puid_injected(self): + """PUID env var must be present in the running container.""" + pod_name = first_pod_name() + out = require_kubectl( + "exec", pod_name, "-n", NAMESPACE, "--", + "sh", "-c", "echo $PUID" + ) + assert out.strip() != "", "PUID env var is not set in the container" + + def test_env_pgid_injected(self): + """PGID env var must be present in the running container.""" + pod_name = first_pod_name() + out = require_kubectl( + "exec", pod_name, "-n", NAMESPACE, "--", + "sh", "-c", "echo $PGID" + ) + assert out.strip() != "", "PGID env var is not set in the container" + + def test_env_tz_injected(self): + """TZ env var must be present in the running container.""" + pod_name = first_pod_name() + out = require_kubectl( + "exec", pod_name, "-n", NAMESPACE, "--", + "sh", "-c", "echo $TZ" + ) + assert out.strip() != "", "TZ env var is not set in the container" + + def test_rollout_persistence(self): + """After a rollout restart the deployment must return to Ready and remain reachable.""" + _rollout_restart() + body = _http_get_ipp() + assert body is not None + + def test_observability_artifact_written(self): + """Lane must emit at least one result artifact that an operator can inspect.""" + artifacts = list(RESULTS_DIR.iterdir()) + assert artifacts, ( + f"No artifacts written to {RESULTS_DIR}; lane produced no observable output" + ) + + @pytest.mark.skip(reason="USB printer device access requires hardware fixture — tracked in issue #67") + def test_usb_printer_device_passthrough(self): + """USB printer hostPath passthrough: deferred to issue #67 (print-device lane).""" + + @pytest.mark.skip(reason="avahi/mDNS LAN discovery requires hardware fixture — tracked in issue #67") + def test_avahi_lan_discovery(self): + """avahi mDNS LAN discovery: deferred to issue #67 (print-device lane)."""