feat: add deployer host installer (#22) - #69
Conversation
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 73f45a1a10
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed8c5e6351
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 777f8c534c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 19a0c4e200
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37d29628fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 773c3ae74c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61b3d0eace
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5ddea4d953
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: deb781caef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Permit the conventional /var/lock -> /run/lock symlink layout instead of rejecting the canonical parent path - Write a structured failed audit record through fd 8 for every failure after approval consumption, including post-adapter snapshot/publication failures, with the failing phase recorded - Reject unknown approval provider values instead of silently skipping GitHub capability validation - Validate the retained last-known-good state schema and cross-match it with the retained policy before rollback starts - Allow --repair to restore a missing or drifted owned active policy by skipping only the untrusted old-policy health probe while retaining candidate validation and post-activation health gates - Fix the repair transaction-failure fixture to fail the candidate health call under the new single-health-call sequence and assert the exact call count
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0b62de8ee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ded stop - The runtime retires the validated pre-adapter incumbent snapshot, never an adapter-writable pointer read after the adapter ran. - The controller installer rejects hosts carrying deployer units or state, preserving the role boundary in both install orders. - The deploy unit's stop timeout is 30s so a hung adapter cannot hold the lock and inhibitor past the 45-minute deployment bound.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 886a95d4b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ring, shortcut metadata, local Docker - The controller rollback path also rejects deployer hosts. - deploy_exit restores the incumbent snapshot's content, not only its pointer, when a failed adapter deleted the target directory. - The terminal failure audit record is appended after restoring the audit log under its durable name. - The identical-bytes publication shortcut now requires the incumbent snapshot directory and files to have safe owner, mode, and type. - Installer and runtime force DOCKER_HOST to the local socket and unset DOCKER_CONTEXT before any Docker call.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 16d5851cac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…-file incumbent restore, shortcut metadata, pre-adapter durability - Both installers now flock a shared /run/ci-fleet-role-admission.lock for their whole mutating run, closing the concurrent-install race. - require_maintenance_host applies the same local-Docker sanitization as require_host. - deploy_exit validates and restores each incumbent snapshot file independently. - The publication shortcut requires root 0600 policy and state files. - perform_rollback syncs all rolled-back core boundaries before invoking the adapter that publishes the commit marker.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5c8302c8bf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… preservation - deploy_exit restores incumbent snapshot directory and file modes as well as contents, and repairs audit-log mode drift before appending the failure record. - The runtime keeps a protected copy of the last-known-good pair across the adapter call and restores it before committing success or failure.
|
@codex review Focused exact-head review at ee99bfc. Apply this objective value gate: report only a reproducible documented-workflow failure, a real trust-boundary authorization/credential issue, plausible loss/corruption of deployment/approval/rollback state, or a concrete repository/CI/runtime/configuration-contract violation that prevents use, validation, deployment, or recovery. Treat duplicates, sibling variants covered by one root fix, style, speculative defense-in-depth, forbidden inputs, and out-of-contract durability as non-blocking. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ee99bfc926
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review Focused exact-head confirmation at b58e8d2. Please verify only the two accepted root fixes: retained rollback state survives recursive adapter cleanup of its writable state root, and immutable versioned adapter paths provide a documented rollback-capable adapter upgrade. Apply the objective review-value gate; collapse duplicates and speculative variants. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b58e8d2e66
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review Focused exact-head confirmation at 515b071. Please verify the accepted rollback-state root fix: after recursive adapter cleanup, the retained pair and completed-deployment marker remain exact and an actual rollback succeeds. Apply the objective review-value gate; collapse duplicates and speculative variants. |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Nickfost
left a comment
Nickfost
left a comment
There was a problem hiding this comment.
[lane-a:pr69:publish-pending-dispositions] Publishing 16 previously drafted accepted/fixed evidence replies. Each corresponding Codex finding already has Nickfost +1; current exact-head CI is green and Codex found no major issues at 515b071.
|
Prepared isolated deployer proof completed on exact head Real Debian 13/systemd/Docker-host lifecycle covered install and unchanged rerun, deploy, exact request/approval/checkpoint binding, application-adapter validate/health/deploy/cleanup/rollback operations, immutable source/artifact state, failed cross-tier request preservation, upgrade, drain/refusal/resume, rollback to the prior application/core state, repeated uninstall, independent demo-b deployment, and complete cleanup. The proof exposed Debian’s canonical |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ad48c75c1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # active-operation guard is durably cleared — but only when the new pointer | ||
| # was never published. After publication the new pointer is the truth: the | ||
| # application has changed and restoring the incumbent would falsify state. | ||
| if [[ ${audit_pending:-0} == 1 && -n ${incumbent_pointer:-} && ${snapshot_pointer:-} != "$deployed_current" ]]; then |
There was a problem hiding this comment.
When the adapter successfully changes the application but a pre-publication integrity check subsequently fails—for example, the adapter also replaces audit.log, causing the check at lines 527-529 to abort—snapshot_pointer is still unset, so this handler restores the incumbent pointer even though the application is already on the new version. Fresh evidence beyond the earlier post-publication finding is that successful-adapter failures can occur before publication as well, leaving the recorded deployed state older than the running application; either publish trustworthy new state or roll the application back before restoring the incumbent.
AGENTS.md reference: AGENTS.md:L64-L66
Useful? React with 👍 / 👎.
| incumbent_backup=$(mktemp -d "$state_root/.incumbent.XXXXXX") | ||
| install -m 0600 "$deployed_current/policy.conf" "$deployed_current/state.json" "$incumbent_backup/" |
There was a problem hiding this comment.
When a failing adapter recursively removes /var/lib/ci-fleet-deployer, it also removes this .incumbent backup before deploy_exit can restore the deployed snapshot. On a first deployment, where no last-known-good pair exists yet, this leaves deployed/current, active policy, and install state gone with no supported rollback or repair path. Fresh evidence beyond the retained-pair cleanup thread is that only the last-known-good pair and completed-request marker moved into process memory; the incumbent backup remains under the deleted root, while the regression exercises cleanup only after an upgrade has supplied an alternate rollback pair.
AGENTS.md reference: AGENTS.md:L64-L66
Useful? React with 👍 / 👎.
| if ! policy_adapter_operation "$active_policy" rollback 'last-known-good policy' "$transaction_dir/application-rollback-committed" && [[ ! -f "$transaction_dir/application-rollback-committed" || -L "$transaction_dir/application-rollback-committed" ]]; then | ||
| die 'application adapter rollback failed' |
There was a problem hiding this comment.
When the rollback adapter recursively removes the deployer state root and returns nonzero, it deletes transaction_dir during this call after the installer has already selected the older core, units, policy, and state. The EXIT handler then calls restore_transaction, which treats the missing journal as a successful no-op, while the adapter's nonzero contract means the application has restored its newer pre-call state; the host is therefore left with an older core—or missing state—describing a newer application. Preserve the recovery journal outside adapter reach, such as in unexported process memory, until the rollback is committed.
AGENTS.md reference: AGENTS.md:L64-L66
Useful? React with 👍 / 👎.
| fi | ||
| reject_mixed_role | ||
| credential_reference_safe "${cfg[CREDENTIAL_PROVIDER]}" "${cfg[CREDENTIAL_REF]}" 'candidate policy' | ||
| run_verified_adapter "$config" "${cfg[ADAPTER_PATH]}" "${cfg[ADAPTER_SHA256]}" validate >/dev/null || die 'candidate adapter validation failed' |
There was a problem hiding this comment.
When a candidate adapter writes to the CI_FLEET_DEPLOYER_CONFIG path during validate and returns zero, this call leaves the installer using attacker- or adapter-modified bytes: line 1172 later copies that same mutable snapshot into active-policy.conf, while write_state still uses the values parsed before the adapter ran. A change to fields that candidate health does not cross-check, such as CORE_REF or SOURCE_COMMIT, can therefore commit a policy/state pair that immediately fails runtime and installer consistency checks. Preserve an immutable copy or verify the snapshot digest after validation before activating it.
AGENTS.md reference: AGENTS.md:L64-L66
Useful? React with 👍 / 👎.
| for unit in ci-fleet-health.service ci-fleet-health.timer ci-fleet-reconcile.service ci-fleet-reconcile.timer ci-fleet-cleanup.service ci-fleet-cleanup.timer ci-fleet-drift.service ci-fleet-drift.timer actions.runner.service; do | ||
| [[ ! -e "$systemd_root/$unit" && ! -L "$systemd_root/multi-user.target.wants/$unit" && ! -L "$systemd_root/timers.target.wants/$unit" ]] || block 'ordinary CI controller or runner state is present' |
There was a problem hiding this comment.
When ci-fleet-status-receiver.service and /etc/ci-fleet-status are already present on an otherwise Docker-clean host, this role check passes because it recognizes only ordinary-controller and runner artifacts. The receiver runbook defines a dedicated host and stores per-controller signing keys plus the read token there, all of which become readable to the root deployer adapter if installation proceeds. A repo-wide search also finds no reciprocal deployer check or shared role-admission lock in install-status-receiver.sh, so installing the receiver second has the same result; reject receiver state in both installers and serialize their role admission.
AGENTS.md reference: AGENTS.md:L34-L34
Useful? React with 👍 / 👎.
1 participant
Summary
--check,--install,--upgrade,--repair,--rollback,--drain, and--uninstallmodesCloses #22
Acceptance matrix
scripts/install-deployer.shrequires exactly one mode and--configscripts/test-install-deployer.sh: missing/duplicate mode, check/install/upgrade/repair/rollback/drain/uninstallrequire_hostsupports Debian 12/13 and Ubuntu 22.04/24.04; Compose is conditionalKEY=valueparser; unknown, duplicate, malformed, missing, and ambiguous fields failscripts/test-deployer-units.shruns realsystemd-analyze verifyin a temporary rootsystemd-inhibitREPORTline carries action, result, environment, target, version, digest, health, changed, rollback, and next actionCHANGED,NO_CHANGE,BLOCKED,FAILED, rollback-available, and canary checksTrust boundaries
The deployer is a dedicated role. It never registers a runner and does not receive pull-request jobs. Ordinary workers/controllers receive no deployment credentials, deployer Docker socket, production authority, or deployment route. The installer rejects mixed-role hosts and unrelated Docker state instead of adopting it.
The systemd runtime runs as root because the Docker socket is root-equivalent. Units apply
NoNewPrivileges,ProtectSystem=strict, private temporary storage, explicit writable paths, and bounded operation timeouts, but do not pretend Docker access is unprivileged. The application adapter is a root-owned mode-0700 regular file pinned by SHA-256. Core defines its operation/timeout/redaction contract; application-specific staging, deployment, health, rollback, credential resolution, and exact-label cleanup remain application-owned.Secret values are never CLI fields. File references stay under the protected configuration tree; external references are resolved by the pinned adapter. Reports and audit logs contain only validated identities, commits, digests, and approval metadata. Network hosts are fed to DNS/HTTPS probes over standard input rather than command arguments.
GitHub-native Environment protection is optional.
github-environmentrequires separate exact-head capability evidence.manual-exact-headandexternal-exact-headprovide fail-closed alternatives for private repositories where the available GitHub plan does not supply the needed protection. A credential store by itself is never treated as approval.Operator examples
On a prepared isolated Linux deployer host, after creating the fictional root-owned configuration/evidence/credential files described in
docs/DEPLOYER-HOST.md:Rollback restores the last-known-good active state but deliberately leaves operator-owned desired configuration/evidence untouched; its report names the required reconciliation step before check.
Validation
scripts/test-install-deployer.shscripts/test-deployer-units.shshellcheck scripts/install-deployer.sh scripts/deployer-runtime.sh scripts/test-install-deployer.sh scripts/test-deployer-units.shscripts/validate.shpython3 scripts/scan_committed_secrets.pygit diff --checkNo host installation, Docker/systemd mutation, runner registration, infrastructure change, credential access, or production deployment was performed. Tests use an explicit temporary-root seam and mocked host commands. A prepared isolated deployment host still needs an operator proof of host prerequisites, application-adapter correctness, target policy, approval evidence, credentials, checkpoint evidence, deployment health, and rollback behavior before use.
Rollback and compatibility
The installer retains hashed core releases, active and last-known-good policy/state, audit logs, and host-local credentials. Failed candidate validation or activation restores the previous units, timer state, policy, state, and release pointer. Uninstall removes only the activation pointer and owned units/timers; retained data requires a separate explicit operator decision.
The public contract is application-neutral and Linux/Bash-only. Existing CI worker/controller behavior is unchanged.