Add isolated test-environment host installer

docs/OPERATOR-WORKFLOWS.md

@@ -19,7 +19,7 @@ Use this page to choose an outcome, confirm the responsible role, and either fol
 | Authorize and onboard an application project | Current, experimental | Application repository, private configuration, and GitHub runner-group policy | Yes; no host mutation | Follow [adding a project](ADDING-A-PROJECT.md), then the [migration procedure](MIGRATING-EXISTING-CI.md). |
 | Create controller GitHub credentials and runner-group prerequisites | Current manual procedure; automation in review | GitHub web UI and the target host | Yes | Follow [GitHub App setup](GITHUB-APP-SETUP.md) and the [runner-group procedure](LIVE-PILOT.md#2-create-the-organization-runner-group). [Issue #27](https://github.com/RandomDevelopment/ci-fleet/issues/27) and [PR #75](https://github.com/RandomDevelopment/ci-fleet/pull/75) track automated bootstrap and its required organization-owner evidence; do not use it from the default branch yet. |
 | Run an isolated first-job proof | Prototype only | GitHub Actions and one isolated worker host | Yes, transiently | Follow the [live pilot](LIVE-PILOT.md). Stop if the matching job queue is not proven empty. |
-| Install a persistent test/staging environment host | In review | Separate test host | N/A on the default branch | [Issue #23](https://github.com/RandomDevelopment/ci-fleet/issues/23) and [PR #76](https://github.com/RandomDevelopment/ci-fleet/pull/76) track the installer and its required real-host evidence. Do not install it from the default branch yet. |
+| Install a persistent test/staging environment host | In review | Separate test host | N/A on the default branch | Follow [test-environment host](TESTER-HOST.md); [issue #23](https://github.com/RandomDevelopment/ci-fleet/issues/23) and [PR #76](https://github.com/RandomDevelopment/ci-fleet/pull/76) track the installer and prepared-host evidence. Do not install it from the default branch yet. |
 | Install a deployment host | In review | Separate deployment host | N/A on the default branch | [Issue #22](https://github.com/RandomDevelopment/ci-fleet/issues/22) and [PR #69](https://github.com/RandomDevelopment/ci-fleet/pull/69) track the installer and its required real-host evidence. Do not install it from the default branch yet. |
 | Deploy to production | Not production-ready | Separate production boundary | N/A | Stop. The project status and production evidence do not authorize this workflow. |
 | Publish or update from the standalone configuration template | Planned release path | Management workstation and a private configuration repository | Yes | Use the template vendored in the exact reviewed engine commit today. A standalone immutable release/compatibility signal remains [open](https://github.com/RandomDevelopment/ci-fleet-config-template/issues/12). |

docs/README.md

@@ -15,6 +15,7 @@ New operator? Use [Operator workflows](OPERATOR-WORKFLOWS.md) to select a suppor
 | Decide whether it fits my infrastructure | [Architecture](ARCHITECTURE.md) and the root [README](../README.md) |
 | Try the experimental implementation safely | [Live pilot runbook](LIVE-PILOT.md) |
 | Add another Docker host, VM, computer, or VPS | [Adding a host](ADDING-A-HOST.md) |
+| Install an isolated deployed-application test host | [Test-environment host](TESTER-HOST.md) |
 | Manage controller configuration from Git | [Git-authored controller desired state](DESIRED-STATE.md) |
 | Create the controller GitHub App identity | [GitHub App setup](GITHUB-APP-SETUP.md) |
 | Add a private project to the shared runner pool | [Adding a project](ADDING-A-PROJECT.md) |

docs/TESTER-HOST.md

@@ -0,0 +1,110 @@
+# Test-environment host
+
+Status: repository implementation complete; prepared isolated-host acceptance remains required.
+
+This role runs persistent or expiring deployed application test environments. It is deliberately separate from ephemeral CI workers and production deployers. It cannot register ordinary CI runners or promote production releases.
+
+## Boundary
+
+A tester host accepts only:
+
+- a reviewed `ci-fleet` source commit for the tester service;
+- root-owned host configuration and environment declarations;
+- application images addressed by an immutable `sha256` digest;
+- environment secrets stored below that environment's fixed host-local secret directory.
+
+It rejects mutable images, public port binds, host bind mounts, external/unscoped Docker resources, custom volume drivers/options, privileged containers, added capabilities, host or shared namespaces, Docker API access, global container names, and credentials outside the environment secret boundary. Compose environment variables, env files, and configs are forbidden credential channels; use only fixed mode-`0600` Compose secrets. Every service must be read-only, drop all capabilities, and set `no-new-privileges=true`. The validated rendered Compose model is copied into protected runtime state before activation, so partial starts remain tracked and later cleanup does not depend on a mutable or deleted source definition. Test identity, networks, storage, routes, domains, data, and credentials must have no production authority. Host/network isolation is an external acceptance gate, not something this repository-only change can prove.
+
+## Prepare host-local configuration
+
+The installer supports only `/etc/ci-fleet-tester/tester.env`. Create it and all protected directories as root; never commit them:
+
+```bash
+sudo install -d -m 0700 \
+  /etc/ci-fleet-tester \
+  /etc/ci-fleet-tester/environments \
+  /etc/ci-fleet-tester/definitions \
+  /etc/ci-fleet-tester/secrets
+sudo install -m 0600 /dev/null /etc/ci-fleet-tester/tester.env
+```
+
+Example non-secret settings:
+
+```text
+CI_FLEET_TESTER_DEFAULT_TTL_SECONDS=86400
+CI_FLEET_TESTER_MAX_ENVIRONMENTS=20
+CI_FLEET_TESTER_DISK_WARN_PERCENT=80
+CI_FLEET_TESTER_NETWORK_PROBE_HOST=tester-probe.invalid
+CI_FLEET_TESTER_HTTPS_PROBE_URL=https://tester-probe.invalid/health
+CI_FLEET_TESTER_ISOLATION_ACK=test-only-no-production-authority
+```
+
+Set both probe values to a test-only host whose DNS resolution and HTTPS HEAD response exercise the intended local proxy path without carrying credentials. The acknowledgement is required but is not proof: an authorized operator must still verify that the prepared host has no production identity or network authority.
+
+For environment `example-preview`, create root-owned mode-`0600` `/etc/ci-fleet-tester/environments/example-preview.env`:
+
+```text
+CI_FLEET_TESTER_PROJECT=example-project
+CI_FLEET_TESTER_OWNER=example-owner
+CI_FLEET_TESTER_COMPOSE_FILE=/etc/ci-fleet-tester/definitions/example-preview.yaml
+CI_FLEET_TESTER_EXPIRES_AT=REVIEWED_FUTURE_UNIX_TIME
+CI_FLEET_TESTER_ROUTE_SERVICE=web
+CI_FLEET_TESTER_ROUTE_PORT=18080
+```
+
+The Compose file is root-owned mode `0644` and may contain no credential value. Each image must use `registry/path@sha256:REVIEWED_64_HEX_DIGEST`. Exactly one route is published, on loopback only, at the declared port. Compose-generated network and volume names must remain below `ci-fleet-test-<environment>_...`; explicit external names are rejected.
+
+If credentials are required, create `/etc/ci-fleet-tester/secrets/example-preview` as root-owned mode `0700`, put only test-scope regular files there as root-owned mode `0600`, and reference them through Compose `secrets.file`. Symlinks, external secrets, production credentials, environment-variable secret transport, and files outside that exact directory are unsupported.
+
+## Fresh install or repair
+
+Use a clean reviewed checkout at the exact commit:
+
+```bash
+ref=$(git rev-parse 'HEAD^{commit}')
+sudo ./scripts/install-tester.sh --install \
+  --config /etc/ci-fleet-tester/tester.env \
+  --ref "$ref"
+```
+
+The command fails before mutation unless it sees Debian 12 or newer, root, the local default Docker context/socket/root, Compose v2, required generic tools, protected paths, and Docker storage below 80%. It stages an immutable source release, validates it, switches the `current` symlink, installs health and expiration timers, and verifies both configuration and active environments. Repeating the same command is idempotent.
+
+## Environment lifecycle
+
+The runtime command is the one interface for create/update, inspect, reset, and removal:
+
+```bash
+sudo /opt/ci-fleet-tester/tester-runtime --converge --environment example-preview
+sudo /opt/ci-fleet-tester/tester-runtime --inspect --environment example-preview
+sudo /opt/ci-fleet-tester/tester-runtime --reset --environment example-preview
+sudo /opt/ci-fleet-tester/tester-runtime --remove --environment example-preview
+```
+
+`--converge` validates the full resolved Compose model before `up --wait`. `--reset` removes only that exact Compose project and its volumes, then recreates it from the approved definition and digest. `--remove` uses the same scoped `compose down --volumes`; no global Docker prune is used. State reports only environment/project/owner, loopback route, expiry, source revision, timestamps, and health—not Compose environment values or secret content.
+
+`ci-fleet-tester-health.timer` checks every five minutes. `ci-fleet-tester-cleanup.timer` checks expiration every fifteen minutes. Expired environments are removed through the same scoped path. Disposable environment data is intentionally removed on reset/expiry. Reviewed definitions and fixture sources belong outside runtime state and need ordinary configuration backups; credentials and disposable volumes are never backed up by this service.
+
+## Upgrade, validation, and rollback
+
+```bash
+ref=$(git rev-parse 'HEAD^{commit}')
+sudo ./scripts/install-tester.sh --upgrade --config /etc/ci-fleet-tester/tester.env --ref "$ref"
+sudo ./scripts/install-tester.sh --check --config /etc/ci-fleet-tester/tester.env
+sudo ./scripts/install-tester.sh --rollback --config /etc/ci-fleet-tester/tester.env
+```
+
+Upgrade validates the candidate before activation and restores the complete previous release if post-switch checks fail. A successful switch records only the previous complete source revision as last known good. Rollback changes the tester service release; it does not rewrite an application's immutable image digest or reset environment data.
+
+## Removal
+
+Remove every environment explicitly, verify no state remains, then uninstall:
+
+```bash
+sudo ./scripts/install-tester.sh --uninstall --config /etc/ci-fleet-tester/tester.env
+```
+
+Uninstall fails while any managed environment exists. It removes only tester units and immutable service releases. Host configuration, definitions, and secrets remain for explicit operator disposition; the script never guesses which credential may be deleted.
+
+## External acceptance gate
+
+Before this draft can merge, an authorized operator must use a prepared isolated Debian Docker host with test-only identity/network/storage/domain boundaries and provide evidence for: fresh install, unchanged second install, immutable application converge, health, scoped reset, expiration cleanup, upgrade, failed-upgrade restoration, rollback, and removal. No production credentials, database, network authority, ordinary CI runner, or live production system may participate.

host/systemd/ci-fleet-tester-cleanup.service

@@ -0,0 +1,20 @@
+[Unit]
+Description=Remove expired ci-fleet test environments
+After=docker.service
+Requires=docker.service
+ConditionPathExists=/etc/ci-fleet-tester/tester.env
+
+[Service]
+Type=oneshot
+ExecStart=/opt/ci-fleet-tester/tester-runtime --cleanup
+User=root
+Group=root
+UMask=0077
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectHome=yes
+ProtectSystem=strict
+ReadWritePaths=/var/lib/ci-fleet-tester -/run/lock/ci-fleet-tester
+RestrictAddressFamilies=AF_UNIX
+LockPersonality=yes
+MemoryDenyWriteExecute=yes

host/systemd/ci-fleet-tester-cleanup.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Expire abandoned ci-fleet test environments
+
+[Timer]
+OnBootSec=5m
+OnUnitActiveSec=15m
+Persistent=true
+RandomizedDelaySec=1m
+Unit=ci-fleet-tester-cleanup.service
+
+[Install]
+WantedBy=timers.target

host/systemd/ci-fleet-tester-health.service

@@ -0,0 +1,20 @@
+[Unit]
+Description=Validate ci-fleet test environments
+After=docker.service
+Requires=docker.service
+ConditionPathExists=/etc/ci-fleet-tester/tester.env
+
+[Service]
+Type=oneshot
+ExecStart=/opt/ci-fleet-tester/tester-runtime --health
+User=root
+Group=root
+UMask=0077
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectHome=yes
+ProtectSystem=strict
+ReadWritePaths=/var/lib/ci-fleet-tester -/run/lock/ci-fleet-tester
+RestrictAddressFamilies=AF_UNIX
+LockPersonality=yes
+MemoryDenyWriteExecute=yes

host/systemd/ci-fleet-tester-health.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Check ci-fleet test environments every five minutes
+
+[Timer]
+OnBootSec=2m
+OnUnitActiveSec=5m
+Persistent=true
+RandomizedDelaySec=30s
+Unit=ci-fleet-tester-health.service
+
+[Install]
+WantedBy=timers.target

scripts/fixtures/fake-tester-docker.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+printf '%s\n' "$*" >>"${FAKE_TESTER_DOCKER_LOG:?}"
+[[ -z ${FAKE_TESTER_EVENT_LOG:-} ]] || printf 'docker %s\n' "$*" >>"$FAKE_TESTER_EVENT_LOG"
+if [[ $1 == context && $2 == show ]]; then printf 'default\n'; exit 0; fi
+if [[ $1 == info ]]; then printf '%s\n' "${FAKE_TESTER_DOCKER_ROOT:?}"; exit 0; fi
+if [[ $1 == ps ]]; then [[ ${FAKE_TESTER_PS_FAIL:-0} != 1 ]] || exit 9; printf 'fixture-container-id\n'; exit 0; fi
+if [[ $1 == inspect ]]; then
+  if [[ " $* " == *' --size '* ]]; then printf '1024\n'; else printf '%s\n' "${FAKE_TESTER_CONTAINER_STATE:-running healthy}"; fi
+  exit 0
+fi
+if [[ $1 == volume && $2 == ls ]]; then [[ ${FAKE_TESTER_VOLUME_LS_FAIL:-0} != 1 ]] || exit 9; printf 'fixture-volume\n'; exit 0; fi
+if [[ $1 == volume && $2 == inspect ]]; then printf '%s\n' "${FAKE_TESTER_VOLUME_ROOT:?}"; exit 0; fi
+if [[ $1 == compose && $2 == version ]]; then printf 'Docker Compose version v2.fixture\n'; exit 0; fi
+if [[ $1 == compose && $2 == up && ${3:-} == --help ]]; then [[ ${FAKE_TESTER_NO_WAIT_TIMEOUT:-0} != 1 ]] && printf '%s\n' '  --wait-timeout int'; exit 0; fi
+if [[ $1 != compose ]]; then exit 2; fi
+shift
+project=
+while (($#)); do
+  case $1 in
+    -p) project=$2; shift 2 ;;
+    -f) shift 2 ;;
+    config|up|down|ps) operation=$1; shift; break ;;
+    *) shift ;;
+  esac
+done
+case ${operation:-} in
+  config)
+    digest=$(printf 'a%.0s' {1..64})
+    privileged=false; read_only=true; host_ip=127.0.0.1; image="registry.example/example/app@sha256:$digest"; network_name="${project}_default"; secrets='{}'; service_extra=; top_extra=; volume_extra=; network_extra=; security='no-new-privileges:true'
+    case ${FAKE_TESTER_POLICY:-valid} in
+      mutable) image=registry.example/example/app:latest ;;
+      privileged) privileged=true ;;
+      bind) volume='{"type":"bind","source":"/","target":"/host"}' ;;
+      broad-port) host_ip=0.0.0.0 ;;
+      external-network) network_name=shared ;;
+      environment) service_extra=',"environment":{"TOKEN":"example"}' ;;
+      configs) service_extra=',"configs":[{"source":"credential"}]'; top_extra=',"configs":{"credential":{"file":"/tmp/example"}}' ;;
+      use-api-socket) service_extra=',"use_api_socket":true' ;;
+      namespace-share) service_extra=',"network_mode":"service:other"' ;;
+      false-nnp) security='no-new-privileges:false' ;;
+      custom-volume) volume_extra=',"driver":"local"' ;;
+      volumes-from) service_extra=',"volumes_from":["container:other:rw"]' ;;
+      custom-network) network_extra=',"driver":"macvlan","driver_opts":{"parent":"eth0"}' ;;
+      replicas) service_extra=',"deploy":{"replicas":2}' ;;
+      lifecycle-hook) service_extra=',"post_start":[{"command":"true","privileged":true}]' ;;
+      gpu) service_extra=',"gpus":"all"' ;;
+      deploy-device) service_extra=',"deploy":{"resources":{"reservations":{"devices":[{"capabilities":["gpu"]}]}}}' ;;
+      build) service_extra=',"build":{"context":"/"}' ;;
+      changed-model) service_extra=',"pull_policy":"always"' ;;
+      unconfined) security='no-new-privileges:true","seccomp=unconfined' ;;
+      external-links) service_extra=',"external_links":["other-environment-db:db"]' ;;
+      userns-host) service_extra=',"userns_mode":"host"' ;;
+      cgroup-host) service_extra=',"cgroup":"host"' ;;
+      uts-host) service_extra=',"uts":"host"' ;;
+      remote-logging) service_extra=',"logging":{"driver":"syslog","options":{"syslog-address":"tcp://example.invalid:514"}}' ;;
+      interpolation) [[ -z ${TOKEN:-} ]] || service_extra=$(printf ',"command":["app","--token=%s"]' "$TOKEN") ;;
+      valid-secret|outside-secret) secrets=$(printf '{"credential":{"file":"%s"}}' "${FAKE_TESTER_SECRET_FILE:?}") ;;
+    esac
+    volume=${volume:-'{"type":"volume","source":"data","target":"/data"}'}
+    printf '{"services":{"web":{"image":"%s","privileged":%s,"read_only":%s,"cap_drop":["ALL"],"security_opt":["%s"],"volumes":[%s],"ports":[{"host_ip":"%s","published":%s,"target":8080,"protocol":"tcp"}]%s}},"networks":{"default":{"name":"%s"%s}},"volumes":{"data":{"name":"%s_data"%s}},"secrets":%s%s}\n' \
+      "$image" "$privileged" "$read_only" "$security" "$volume" "$host_ip" "${FAKE_TESTER_ROUTE_PORT:-18080}" "$service_extra" "$network_name" "$network_extra" "$project" "$volume_extra" "$secrets" "$top_extra"
+    ;;
+  up) [[ ${FAKE_TESTER_UP_FAIL:-0} != 1 ]] ;;
+  down) [[ ${FAKE_TESTER_DOWN_FAIL:-0} != 1 ]] ;;
+  ps) [[ ${FAKE_TESTER_UNHEALTHY:-0} == 1 ]] || printf 'fixture-container-id\n' ;;
+  *) exit 2 ;;
+esac