Skip to content

Add isolated test-environment host installer - #76

Open
Nickfost wants to merge 17 commits into
mainfrom
feat/issue-23-tester-host
Open

Add isolated test-environment host installer#76
Nickfost wants to merge 17 commits into
mainfrom
feat/issue-23-tester-host

Conversation

@Nickfost

Copy link
Copy Markdown
Member

Closes #23

Summary

  • add idempotent install/check/upgrade/uninstall workflows for a root-owned isolated tester role
  • add policy-validated digest-pinned Compose environments, bounded TTL/disk cleanup, loopback-only routing, and secret isolation
  • add health/readiness checks, systemd timers, examples, tests, and operator documentation

Validation

  • scripts/test-install-tester.sh (TESTER_INSTALLER_TESTS_OK)
  • Bash syntax and ShellCheck
  • documentation links, diff/trailer/identity, and committed-secret scan
  • scripts/validate.sh reached INSTALLER_TESTS_OK and STATUS_RECEIVER_INSTALL_TESTS_OK; final Docker smoke was omitted because this repository-only environment has no Docker CLI/daemon (docker: command not found)

External gate

Draft: issue acceptance still requires install/upgrade/rollback/cleanup evidence on a prepared Debian test host. This run did not touch a live host, Docker daemon, systemd, network, secret, test environment, or production system.

@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head b86cb94 for isolation boundaries, idempotence, rollback/state preservation, remote-Docker rejection, cleanup scope, and mocked acceptance coverage.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b86cb94df9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/tester-runtime.sh
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/install-tester.sh
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/test-install-tester.sh
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please confirm exact head a75e2fa69afe6bff6b4f6a2b3f211b80495bd65b after the accepted lifecycle, policy, rollback, health, and release-integrity fixes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a75e2fac15

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/tester-runtime.sh
Comment thread scripts/install-tester.sh
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/install-tester.sh Outdated
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please confirm exact head f36fa02c3188de680087913e34bcb4f739641bd6; this only makes read-only release tamper/teardown tests work under non-root CI, with the full tester suite passing.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f36fa02c6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head 196b121 after the accepted Compose isolation, release recovery, lifecycle locking, teardown, and regression fixes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 196b121df6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head ca39b25 after the accepted immutable-build, model-replacement, teardown, lifecycle-serialization, and hard-link fixes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca39b25234

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/tester-runtime.sh
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/install-tester.sh
Comment thread scripts/tester-runtime.sh Outdated
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head b4736e5 after the accepted cleanup containment, incumbent restoration, reset locking, and atomic model-preservation fixes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b4736e5e1a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/install-tester.sh
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head 841406c after the accepted timer-quiescence, confinement-option, and bounded-teardown fixes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 841406c946

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/tester-runtime.sh
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head d88df91 after the accepted cross-link, timer-restoration, namespace, interpolation, and activation-timeout fixes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d88df91e1d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/tester-runtime.sh
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/install-tester.sh
Comment thread scripts/tester-runtime.sh
Comment thread scripts/install-tester.sh
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head bec7c76 after the accepted include, UTS, stable-launcher, logging, and lock-directory fixes.

@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head 7c0e0e3. This follow-up only corrects the lifecycle-lock regression to use the new dedicated lock path; focused mocked validation passes.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 7c0e0e34b6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Nickfost

Copy link
Copy Markdown
Member Author

Prepared isolated tester proof completed on exact head c600b8fbd73a5625b838464ebbea4590b4587e8b: Debian 13, Docker Engine 29.7.2, Compose v5.4.0. Real lifecycle covered preflight; reviewed-predecessor install and unchanged rerun; final-head upgrade/check; rollback and re-upgrade; rejected unsafe Compose; two healthy isolated services per environment; digest pinning; loopback routes; secret-file hash equivalence without disclosure; read-only/capability/socket/namespace/network/volume boundaries; stable expiry; scoped reset; expiry cleanup; uninstall refusal with managed environments; repeated removal/cleanup; and repeated uninstall.

The live proof exposed one repository-owned defect: a second uninstall failed when systemd units were already absent. 3c56ef7ee32423cfe093fe30f1faaa0c0fa4746b fixes the shared teardown path and adds a regression; scripts/test-install-tester.sh, Bash syntax, ShellCheck, diff check, and committed-secret scan pass. Final cleanup proves zero containers, images, volumes, custom networks, runner processes/units, credentials, or ci-fleet tester state. No production or ordinary-runner resource participated.

@Nickfost
Nickfost marked this pull request as ready for review August 17, 2026 02:24

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c600b8fbd7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/tester-launcher.sh
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/install-tester.sh Outdated
Comment thread scripts/tester-runtime.sh Outdated
Comment thread scripts/install-tester.sh
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head e7640e2 after the six accepted exact-head findings: reboot-safe/symlink-safe locking, escaped-key include rejection, transactional last-known-good failure recovery, checked Docker inventories, and Compose capability preflight. Focused tester validation passes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7640e2362

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/tester-runtime.sh
Comment on lines +116 to +118
key=r'(?:!!str[ \t]+)?(?:include|"include"|\x27include\x27)[ \t]*:'
escaped_key=r'"[^"\n]*\\[^"\n]*"[ \t]*:'
if re.search(r'(?m)^[ \t]*'+key,text) or re.search(r'[,{][ \t]*'+key,text) or re.search(escaped_key,text): raise SystemExit('Compose include or escaped mapping key is forbidden')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject explicit-key Compose includes

Fresh evidence after the accepted escaped-key fix is YAML's explicit mapping form: a source containing ? "include" followed by : [{path: ..., env_file: /host/credential}] does not match either regex because the key and colon are on separate lines. Compose nevertheless parses the decoded top-level include and processes its interpolation file before emitting the rendered JSON, so credential values can be embedded in commands or labels with no remaining include or env_file field for the later policy to reject. Parse the YAML key or conservatively reject explicit mapping keys before rendering.

AGENTS.md reference: AGENTS.md:L33-L34

Useful? React with 👍 / 👎.

PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/ci-fleet-tester -/run/lock/ci-fleet-tester

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Make the systemd lock path reboot-creatable

Fresh evidence beyond the accepted launcher recreation fix is this systemd sandbox entry: after reboot, when the volatile directory is absent, ProtectSystem=strict makes /run/lock read-only, while the leading - on ReadWritePaths means the missing exception is ignored. The launcher's mkdir therefore cannot create the directory when either timer starts, leaving expiration cleanup and health checks broken until an unsandboxed command recreates it; create it with RuntimeDirectory= or grant a writable existing parent.

AGENTS.md reference: AGENTS.md:L35-L35

Useful? React with 👍 / 👎.

Comment thread scripts/install-tester.sh
Comment on lines +140 to +142
for unit in "${units[@]}"; do [[ ! -e $systemd_dir/$unit ]] || present=1; done
if ((present)); then systemctl disable --now "${timers[@]}" >/dev/null 2>&1 || return 1
else systemctl disable --now "${timers[@]}" >/dev/null 2>&1 || true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove partial unit sets without requiring both timers

If install_units fails after moving only a prefix of the unit files during a fresh activation, present is true but one of the timer units can still be absent. This call passes both names to systemctl disable; checked with systemctl 255, a missing named unit makes the command return nonzero, so remove_units exits here without deleting any partial files, and every subsequent uninstall repeats the same failure. Disable only installed timers or tolerate the not-found result while still surfacing genuine stop failures.

AGENTS.md reference: AGENTS.md:L66-L66

Useful? React with 👍 / 👎.

Comment thread scripts/tester-runtime.sh
Comment on lines +236 to +238
expected=$(python3 -c 'import json,sys; print(len(json.load(open(sys.argv[1]))["services"]))' "$compose")
mapfile -t containers < <(docker compose -p "$(project_name "$id")" -f "$compose" ps -q)
[[ ${#containers[@]} == "$expected" ]] || status=unhealthy

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject inactive profiled services

When a validated definition assigns one service a Compose profiles entry, the unprofiled compose up invocation does not start that service, but the rendered configuration still includes it here. Inspection therefore expects more containers than Compose created and permanently reports the otherwise successful environment as unhealthy; if the profiled service is the declared route, convergence can also report a route that was never started. Reject profiles or calculate the expected service set using the same profile selection as activation.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an idempotent Bash installer for test environment hosts

1 participant