From f59bdaae62b523d3757a5fa061d9a123146b19c2 Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sat, 15 Aug 2026 22:23:18 -0500 Subject: [PATCH 01/10] feat: automate GitHub App bootstrap --- docs/GITHUB-APP-SETUP.md | 165 +++++------ scripts/bootstrap-github-callback.py | 126 +++++++++ scripts/bootstrap-github.sh | 353 ++++++++++++++++++++++++ scripts/fixtures/fake-bootstrap-curl.py | 60 ++++ scripts/test-bootstrap-github.sh | 111 ++++++++ scripts/validate.sh | 3 + 6 files changed, 725 insertions(+), 93 deletions(-) create mode 100755 scripts/bootstrap-github-callback.py create mode 100755 scripts/bootstrap-github.sh create mode 100755 scripts/fixtures/fake-bootstrap-curl.py create mode 100755 scripts/test-bootstrap-github.sh diff --git a/docs/GITHUB-APP-SETUP.md b/docs/GITHUB-APP-SETUP.md index d32990b..19ada63 100644 --- a/docs/GITHUB-APP-SETUP.md +++ b/docs/GITHUB-APP-SETUP.md @@ -16,104 +16,70 @@ organization, app name, and configuration repository. identity, not a personal account. - Revoking access is removing an installation, not rotating a user's keys. -## 1. Create the app +## Bootstrap on the target host -Organization Settings → Developer settings → GitHub Apps → New GitHub App. - -| Field | Value | -| --- | --- | -| Name | one per controller, e.g. `ci-fleet-` | -| Homepage URL | your organization or fleet repository URL | -| Webhook | disabled — the controller polls; nothing calls it | - -## 2. Grant minimum permissions - -Repository permissions: - -| Permission | Access | Why | -| --- | --- | --- | -| Contents | Read-only | fetch desired-state configuration over HTTPS | - -Organization permissions: - -| Permission | Access | Why | -| --- | --- | --- | -| Self-hosted runners | Read & write | mint runner registration tokens | - -Nothing else. No `write` on contents, no actions, no administration. If the -controller ever needs more, that is a reviewed design change, not a settings -tweak. - -## 3. Generate and transfer the private key - -On the app page: Private keys → Generate a private key. GitHub downloads one -PEM. The normal manual workflow in this guide stores it on the controller's -local filesystem at `/etc/ci-fleet/secrets/github-app.pem`, owned by root with -mode `0600`. - -GitHub's browser download is necessarily present briefly on a controlled -management workstation. Choose a fresh temporary directory outside synchronized, -indexed, and backed-up locations, restrict the downloaded file to the operator -immediately, and transfer it at once. This is transient handling, not approved -long-term credential storage; never claim that the key was absent from the -workstation. - -Before any key bytes arrive, create the controller directory as root-owned mode -`0700` and pre-create the destination as a root-owned regular file with mode -`0600`. Initial setup uses only the active path: +Run the reviewed checkout on the target Linux Docker host. The callback defaults +to loopback. For phone-first use, explicitly choose one private address that the +phone can reach; the script rejects public callback addresses. ```bash -sudo install -d -o root -g root -m 0700 /etc/ci-fleet/secrets -sudo install -o root -g root -m 0600 /dev/null \ - /etc/ci-fleet/secrets/github-app.pem +sudo ./scripts/bootstrap-github.sh \ + --organization example-org \ + --instance example-ci-01 \ + --runner-group example-ci-experimental \ + --allow-repository example-org/example-repo ``` -Verify the directory and destination ownership, type, and mode without reading -the content. Then use an authenticated encrypted channel to stream into that -already secured file; the transfer must not replace it with a default-mode node. -Keep key bytes out of tracing, logs, stdout, process arguments, Git, issues, and -PRs. Compare a SHA-256 digest at both ends without printing file content, then -verify the destination again. - -Delete the workstation copy immediately after authenticated transfer and those -transfer checks succeed, before token, reconciliation, health, or convergence -checks. Stop if transfer verification or local deletion fails. The PEM must -never be committed or printed; see [SECRETS.md](SECRETS.md). - -This manual workflow permits exactly two host-local files: the active path above -and `/etc/ci-fleet/secrets/github-app.next.pem` while rotating. It does not cover -other custom paths, symlinks, or an external secret manager's import, rotation, -or deletion lifecycle. Those cases require provider-specific tested automation. -Do not improvise them from these Markdown examples; -[issue #27](https://github.com/RandomDevelopment/ci-fleet/issues/27) tracks that -automation. - -## 4. Install the app - -App page → Install App → choose the organization → **Only select -repositories**: pick only the private desired-state configuration repository. -Installing on all repositories defeats the permission scoping. - -Record from the installation page URL and app page: - -- **Client ID** (app page, `Iv1...` / `Iv23...`) -- **Installation ID** (the number at the end of the installation URL) - -## 5. Wire the host - -`/etc/ci-fleet/host.env` (root-owned `0600`, never committed): +For phone access, append `--bind PRIVATE_IP --callback-host PRIVATE_IP` using the +same explicitly selected host-local private address; do not publish that address +in Git, logs, or support messages. + +All names above are fictional. The script prints one non-secret local +`REGISTRATION_URL`. Open it, press the single registration button, install the +new App for **only** the requested private repositories, and return to the +terminal. The target host receives and exchanges the temporary code itself. +Neither the code nor any credential is copied through a phone, clipboard, chat, +email, issue, or second computer. + +The bootstrap: + +- requests only `contents: read`, metadata read, and organization self-hosted + runner write permission; +- creates a private, independently revocable App identity for the host; +- verifies callback state and expires the callback after 30–1800 seconds; +- writes the PEM directly to `/etc/ci-fleet/secrets/github-app.pem` and the + client/installation IDs to `/etc/ci-fleet/host.env`, root-owned mode `0600`; +- rejects public, archived, wrong-organization, broader App installation, and + default/broad runner-group access; +- creates a missing selected-repository runner group, but never changes an + existing group whose identity or access differs; +- destroys the conversion code, conversion response, JWTs, installation token, + callback state, and temporary curl configurations on every exit. + +Inspect a request without local writes or GitHub calls: ```bash -CI_FLEET_GITHUB_APP_CLIENT_ID= -CI_FLEET_GITHUB_APP_INSTALLATION_ID= -CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE=/etc/ci-fleet/secrets/github-app.pem +sudo ./scripts/bootstrap-github.sh --dry-run \ + --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental \ + --allow-repository example-org/example-repo ``` +After success, rerun the same command with `--check` to verify App ownership, +permissions, exact selected private repositories, exact runner-group access, +and host-local credential modes without changing the App or runner group. + +Pass `--install --config-repo OWNER/REPO --config-ref REVIEWED_COMMIT` on the +initial live command for direct handoff to the idempotent host installer. Without +it, the redacted final report prints the exact installer shape. Externally +provisioned credentials remain supported; the generic installer never requires +this bootstrap on every run. + The controller exchanges a short-lived JWT signed with the PEM for an installation token at runtime (`scripts/github-app-token.sh`). No token is stored. -## 6. Verify +## Verify The token helper writes a token to stdout. Every verification invocation must redirect stdout to `/dev/null`; exit status alone is the result. @@ -152,21 +118,34 @@ sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh --check-only - 401 on token exchange: `host.env` has the wrong client ID, installation ID, or PEM path. +## Accidental creation rollback + +The bootstrap never replaces or broadens an existing App or runner group. If a +new object was approved accidentally, stop before running the installer. An +organization owner must compare the App slug/ID and group name in the redacted +bootstrap report with GitHub's settings, verify the new group has no runners, +then remove only those exact newly created objects. Preserve and investigate any +pre-existing or mismatched object. App/group deletion and owner approval are live +GitHub-setting mutations and therefore require separate authorization; this +repository command does not automate them. + ## Rotation Use this ordered safety checklist for the normal host-local root-owned PEM workflow. It is a set of gates, not a copy-and-paste shell program. -1. Generate a new GitHub key and transfer, verify, and install it as described - above at the one approved replacement path, +1. Before any key bytes arrive, use a controlled management workstation and + pre-create the one approved replacement path, `/etc/ci-fleet/secrets/github-app.next.pem`. Pre-create it as root-owned - `0600` inside the root-owned `0700` directory before transfer. Keep the old - key and `/etc/ci-fleet/secrets/github-app.pem` active for rollback, and delete - the workstation copy before continuing. + `0600` inside the root-owned `0700` directory. Generate a new GitHub key, + transfer and verify it there, and keep the old key and + `/etc/ci-fleet/secrets/github-app.pem` active for rollback. Delete the + workstation copy immediately after verified transfer and before token, + reconciliation, health, or convergence checks. 2. Update the protected controller identity configuration to select the new PEM. -3. Activate the new key and require the installed manager's token verification, - reconciliation, health check, and installed-state convergence check all to - succeed with the new key. +3. Require new-key activation, reconciliation, health, or convergence checks + all to succeed with the new key, including the installed manager's token + verification and installed-state convergence check. 4. Confirm the controller remains healthy and converged after a fresh check. 5. Only then revoke the old key in GitHub. Remove its exact old controller PEM only after revocation is confirmed. Retain the now-active replacement PEM; diff --git a/scripts/bootstrap-github-callback.py b/scripts/bootstrap-github-callback.py new file mode 100755 index 0000000..708d083 --- /dev/null +++ b/scripts/bootstrap-github-callback.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +import argparse +import html +import http.server +import os +import secrets +import urllib.parse +from typing import cast + + +class Server(http.server.HTTPServer): + organization: str + state: str + manifest: str + output: str + handoff: str + done: bool + + +class Callback(http.server.BaseHTTPRequestHandler): + server_version = "ci-fleet-bootstrap" + sys_version = "" + + @property + def app_server(self) -> Server: + return cast(Server, self.server) + + def log_message(self, _format: str, *_args: object) -> None: + return + + def reply(self, status: int, body: str) -> None: + data = body.encode() + self.send_response(status) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(data))) + self.send_header("Cache-Control", "no-store") + self.send_header("X-Content-Type-Options", "nosniff") + self.send_header("X-Frame-Options", "DENY") + self.send_header("Content-Security-Policy", "default-src 'none'; form-action https://github.com; frame-ancestors 'none'") + self.end_headers() + self.wfile.write(data) + + def do_GET(self) -> None: + parsed = urllib.parse.urlsplit(self.path) + if parsed.path == "/" and not parsed.query: + action = f"https://github.com/organizations/{urllib.parse.quote(self.app_server.organization, safe='')}/settings/apps/new?state={urllib.parse.quote(self.app_server.state, safe='')}" + body = ( + "" + "Register ci-fleet GitHub App" + "

Register this host's ci-fleet App

" + f"
" + f"" + "
" + ) + self.reply(200, body) + return + if parsed.path == "/next" and not parsed.query: + try: + target = open(self.app_server.handoff, encoding="utf-8").read().strip() + except FileNotFoundError: + self.reply(200, "

Preparing the installation approval link…

") + return + if not target.startswith("https://github.com/apps/") or not target.endswith("/installations/new"): + self.reply(500, "Invalid installation target") + return + self.reply(200, f"

App created

Continue to GitHub installation approval

") + self.app_server.done = True + return + if parsed.path != "/callback": + self.reply(404, "Not found") + return + try: + values = urllib.parse.parse_qs(parsed.query, strict_parsing=True) + except ValueError: + self.reply(400, "Invalid callback") + return + if set(values) != {"code", "state"} or any(len(value) != 1 for value in values.values()): + self.reply(400, "Invalid callback") + return + if not secrets.compare_digest(values["state"][0], self.app_server.state): + self.reply(403, "Invalid callback state") + return + code = values["code"][0] + if not code or len(code) > 512 or any(character.isspace() for character in code): + self.reply(400, "Invalid callback") + return + try: + descriptor = os.open(self.app_server.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + except FileExistsError: + self.reply(409, "Callback already consumed") + return + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + handle.write(code) + self.reply(200, "

Registration received

Preparing installation approval. No value needs to be copied.

") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--bind", required=True) + parser.add_argument("--port", required=True, type=int) + parser.add_argument("--organization", required=True) + parser.add_argument("--state", required=True) + parser.add_argument("--manifest", required=True) + parser.add_argument("--output", required=True) + parser.add_argument("--handoff", required=True) + parser.add_argument("--timeout", type=int, default=600) + args = parser.parse_args() + if not 1 <= args.port <= 65535 or not 30 <= args.timeout <= 1800: + parser.error("invalid port or timeout") + server = Server((args.bind, args.port), Callback) + server.organization = args.organization + server.state = args.state + server.manifest = args.manifest + server.output = args.output + server.handoff = args.handoff + server.done = False + server.timeout = 1 + deadline = __import__("time").monotonic() + args.timeout + while __import__("time").monotonic() < deadline and not server.done: + server.handle_request() + server.server_close() + return 0 if server.done else 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh new file mode 100755 index 0000000..756be4e --- /dev/null +++ b/scripts/bootstrap-github.sh @@ -0,0 +1,353 @@ +#!/usr/bin/env bash +set -Eeuo pipefail +umask 077 + +usage() { + cat <<'EOF' +Usage: bootstrap-github.sh [--dry-run|--check] --organization ORG --instance ID + --runner-group GROUP --allow-repository ORG/REPO [--allow-repository ORG/REPO ...] + [--bind ADDRESS --callback-host HOST --port PORT --timeout SECONDS] + [--install --config-repo OWNER/REPO_OR_PATH --config-ref COMMIT] +EOF +} +die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } +note() { printf '%s\n' "$*"; } + +mode=live; organization=; instance=; runner_group=; bind=127.0.0.1; callback_host=127.0.0.1 +port=8765; timeout=600; run_installer=false; config_repo=; config_ref= +repositories=() +while (($#)); do + case $1 in + --dry-run) mode=dry-run; shift ;; + --check) mode=check; shift ;; + --organization) (($# >= 2)) || die '--organization requires a value'; organization=$2; shift 2 ;; + --instance) (($# >= 2)) || die '--instance requires a value'; instance=$2; shift 2 ;; + --runner-group) (($# >= 2)) || die '--runner-group requires a value'; runner_group=$2; shift 2 ;; + --allow-repository) (($# >= 2)) || die '--allow-repository requires a value'; repositories+=("$2"); shift 2 ;; + --bind) (($# >= 2)) || die '--bind requires a value'; bind=$2; shift 2 ;; + --callback-host) (($# >= 2)) || die '--callback-host requires a value'; callback_host=$2; shift 2 ;; + --port) (($# >= 2)) || die '--port requires a value'; port=$2; shift 2 ;; + --timeout) (($# >= 2)) || die '--timeout requires a value'; timeout=$2; shift 2 ;; + --install) run_installer=true; shift ;; + --config-repo) (($# >= 2)) || die '--config-repo requires a value'; config_repo=$2; shift 2 ;; + --config-ref) (($# >= 2)) || die '--config-ref requires a value'; config_ref=$2; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) usage; die "unknown argument: $1" ;; + esac +done +[[ $organization =~ ^[A-Za-z0-9][A-Za-z0-9-]{0,38}$ ]] || die 'invalid organization' +[[ $instance =~ ^[a-z0-9][a-z0-9.-]{0,62}$ ]] || die 'invalid instance identity' +[[ $runner_group =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$ && ${runner_group,,} != default ]] || die 'runner group must be explicit and non-default' +((${#repositories[@]} > 0)) || die 'at least one --allow-repository is required' +[[ $port =~ ^[0-9]+$ && $port -ge 1 && $port -le 65535 ]] || die 'invalid callback port' +[[ $timeout =~ ^[0-9]+$ && $timeout -ge 30 && $timeout -le 1800 ]] || die 'timeout must be 30-1800 seconds' +for repository in "${repositories[@]}"; do + [[ $repository =~ ^${organization}/[A-Za-z0-9_.-]+$ ]] || die "repository must belong to $organization: $repository" +done +mapfile -t repositories < <(printf '%s\n' "${repositories[@]}" | LC_ALL=C sort -u) +((${#repositories[@]} > 0)) || die 'repository allowlist is empty' +if $run_installer; then + [[ $mode == live ]] || die '--install is available only in live mode' + [[ -n $config_repo && $config_ref =~ ^[0-9a-f]{40}$ ]] || die '--install requires --config-repo and a 40-character --config-ref' +fi +for command in bash curl openssl python3 install mktemp stat; do command -v "$command" >/dev/null || die "required command is unavailable: $command"; done + +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +root_prefix=${CI_FLEET_ROOT_PREFIX:-} +if [[ -n $root_prefix && ${CI_FLEET_TESTING:-0} != 1 ]]; then die 'CI_FLEET_ROOT_PREFIX is test-only'; fi +root_path() { printf '%s%s' "$root_prefix" "$1"; } +etc_dir=$(root_path /etc/ci-fleet) +secret_dir=$etc_dir/secrets +pem=$secret_dir/github-app.pem +host_env=$etc_dir/host.env +bootstrap_state=$etc_dir/bootstrap-app.env +temporary=$(mktemp -d) +callback_pid= +expected_uid=0 +[[ ${CI_FLEET_TESTING:-0} != 1 ]] || expected_uid=$(id -u) +cleanup() { + local status=$? + trap - EXIT INT TERM HUP + [[ -z $callback_pid ]] || kill "$callback_pid" >/dev/null 2>&1 || true + rm -rf "$temporary" + exit "$status" +} +trap cleanup EXIT INT TERM HUP + +if [[ $mode != dry-run && ${CI_FLEET_TESTING:-0} != 1 && ${EUID:-$(id -u)} -ne 0 ]]; then die 'run live/check bootstrap as root'; fi +python3 - "$bind" "$callback_host" <<'PY' || die 'callback bind/host must be loopback or private IP addresses' +import ipaddress, sys +addresses=[ipaddress.ip_address(value) for value in sys.argv[1:]] +if addresses[0] != addresses[1]: raise SystemExit(1) +address=addresses[0] +if address.is_unspecified or not (address.is_loopback or address.is_private): raise SystemExit(1) +PY + +write_auth_config() { + local credential=$1 method=$2 url=$3 output=$4 payload=${5:-} + local config=$temporary/curl.$RANDOM.conf + python3 - "$credential" "$method" "$url" "$output" "$payload" "$config" <<'PY' +from pathlib import Path +import sys +credential, method, url, output, payload, target = sys.argv[1:] +token = Path(credential).read_text().strip() +lines = ['silent', 'show-error', 'fail-with-body', f'request = "{method}"', f'url = "{url}"', + f'output = "{output}"', 'header = "Accept: application/vnd.github+json"', + 'header = "X-GitHub-Api-Version: 2022-11-28"', f'header = "Authorization: Bearer {token}"'] +if payload: + lines += ['header = "Content-Type: application/json"', f'data-binary = "@{payload}"'] +Path(target).write_text('\n'.join(lines) + '\n') +PY + chmod 600 "$config" + curl --config "$config" + rm -f "$config" +} + +make_app_jwt() { + local app_id=$1 key=$2 output=$3 now header payload + now=$(date +%s) + header=$(printf '%s' '{"alg":"RS256","typ":"JWT"}' | openssl base64 -A | tr '+/' '-_' | tr -d '=') + payload=$(printf '{"iat":%d,"exp":%d,"iss":"%s"}' "$((now - 60))" "$((now + 540))" "$app_id" | openssl base64 -A | tr '+/' '-_' | tr -d '=') + printf '%s.%s' "$header" "$payload" >"$temporary/jwt-input" + openssl dgst -sha256 -sign "$key" "$temporary/jwt-input" | openssl base64 -A | tr '+/' '-_' | tr -d '=' >"$temporary/jwt-signature" + printf '%s.%s\n' "$(<"$temporary/jwt-input")" "$(<"$temporary/jwt-signature")" >"$output" + chmod 600 "$output" +} + +load_metadata() { + if [[ -f $bootstrap_state && ! -L $bootstrap_state && $(stat -c %a "$bootstrap_state") == 600 && $(stat -c %u "$bootstrap_state") == "$expected_uid" ]]; then + # shellcheck disable=SC1090 + . "$bootstrap_state" + [[ ${CI_FLEET_BOOTSTRAP_ORGANIZATION:-} == "$organization" && ${CI_FLEET_BOOTSTRAP_INSTANCE:-} == "$instance" ]] || die 'existing bootstrap state belongs to another identity' + elif [[ -f $host_env && ! -L $host_env && $(stat -c %a "$host_env") == 600 && $(stat -c %u "$host_env") == "$expected_uid" ]]; then + # shellcheck disable=SC1090 + . "$host_env" + CI_FLEET_GITHUB_APP_ID=${CI_FLEET_GITHUB_APP_CLIENT_ID:-} + CI_FLEET_GITHUB_APP_SLUG= + else + return 1 + fi + [[ ${CI_FLEET_GITHUB_APP_ID:-} =~ ^([0-9]+|(Iv1\.)?[A-Za-z0-9]+)$ && ${CI_FLEET_GITHUB_APP_CLIENT_ID:-} =~ ^(Iv1\.)?[A-Za-z0-9]+$ && ( -z ${CI_FLEET_GITHUB_APP_SLUG:-} || ${CI_FLEET_GITHUB_APP_SLUG:-} =~ ^[a-z0-9-]+$ ) ]] || die 'existing bootstrap state is invalid' + [[ ${CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE:-/etc/ci-fleet/secrets/github-app.pem} == /etc/ci-fleet/secrets/github-app.pem ]] || die 'existing bootstrap PEM path is unsupported' + [[ -f $pem && ! -L $pem && $(stat -c %a "$pem") == 600 && $(stat -c %u "$pem") == "$expected_uid" ]] || die 'existing bootstrap PEM is invalid' +} + +if [[ $mode == dry-run ]]; then + note "DRY_RUN organization=$organization instance=$instance runner_group=$runner_group repositories=${#repositories[@]}" + note "CALLBACK_URL http://$callback_host:$port/" + note 'NO_GITHUB_MUTATION credentials_written=false installer_run=false' + exit 0 +fi + +for directory in "$etc_dir" "$secret_dir"; do + if [[ -e $directory || -L $directory ]]; then + [[ -d $directory && ! -L $directory && $(stat -c %a "$directory") == 700 && $(stat -c %u "$directory") == "$expected_uid" ]] || die "protected directory is unsafe: $directory" + else + install -d -m 0700 "$directory" + fi +done +if ! load_metadata; then + [[ $mode != check ]] || die 'bootstrap state is missing; run live bootstrap first' + [[ ! -e $pem && ! -L $pem && ! -e $host_env && ! -L $host_env ]] || die 'existing credentials lack matching bootstrap identity; no replacement made' + state=$(openssl rand -hex 32) + callback_url="http://$callback_host:$port/callback" + manifest=$(python3 - "$organization" "$instance" "$callback_url" <<'PY' +import json, sys +organization, instance, callback = sys.argv[1:] +print(json.dumps({ + 'name': f'ci-fleet-{organization}-{instance}', 'url': 'https://github.com/RandomDevelopment/ci-fleet', + 'redirect_url': callback, 'public': False, 'hook_attributes': {'active': False, 'url': callback}, + 'default_permissions': {'contents': 'read', 'metadata': 'read', 'organization_self_hosted_runners': 'write'}, + 'default_events': [] +}, separators=(',', ':'))) +PY +) + code_file=$temporary/manifest-code + handoff_file=$temporary/installation-url + python3 "$script_dir/bootstrap-github-callback.py" --bind "$bind" --port "$port" --organization "$organization" --state "$state" --manifest "$manifest" --output "$code_file" --handoff "$handoff_file" --timeout "$timeout" & + callback_pid=$! + note "REGISTRATION_URL http://$callback_host:$port/" + note 'Open the URL, authenticate to GitHub, and approve App creation. Return here without copying any value.' + deadline=$((SECONDS + timeout)) + while [[ ! -f $code_file && $SECONDS -lt $deadline ]]; do + kill -0 "$callback_pid" 2>/dev/null || die 'registration callback stopped before receiving approval' + sleep 1 + done + [[ -f $code_file ]] || die 'registration callback timed out' + conversion=$temporary/conversion.json + conversion_config=$temporary/conversion.conf + python3 - "$code_file" "$conversion" "$conversion_config" <<'PY' +from pathlib import Path +import sys +code_file, output, config = map(Path, sys.argv[1:]) +code = code_file.read_text().strip() +if not code or len(code) > 512 or any(ch.isspace() for ch in code): raise SystemExit(1) +config.write_text('\n'.join(['silent','show-error','fail-with-body','request = "POST"', + f'url = "https://api.github.com/app-manifests/{code}/conversions"', f'output = "{output}"', + 'header = "Accept: application/vnd.github+json"','header = "X-GitHub-Api-Version: 2022-11-28"']) + '\n') +PY + chmod 600 "$conversion_config" + curl --config "$conversion_config" + rm -f "$code_file" "$conversion_config" + python3 - "$conversion" "$pem" "$bootstrap_state" "$organization" "$instance" <<'PY' +import json, os, re, sys +from pathlib import Path +source, pem, state = map(Path, sys.argv[1:4]); organization, instance = sys.argv[4:] +value=json.loads(source.read_text()) +if not isinstance(value.get('pem'),str) or 'PRIVATE KEY' not in value['pem']: raise SystemExit(1) +if not isinstance(value.get('id'),int) or not re.fullmatch(r'(Iv1\.)?[A-Za-z0-9]+',str(value.get('client_id',''))): raise SystemExit(1) +slug=str(value.get('slug','')) +if not re.fullmatch(r'[a-z0-9-]+',slug): raise SystemExit(1) +pem.write_text(value['pem']); os.chmod(pem,0o600) +state.write_text(f'CI_FLEET_BOOTSTRAP_ORGANIZATION={organization}\nCI_FLEET_BOOTSTRAP_INSTANCE={instance}\nCI_FLEET_GITHUB_APP_ID={value["id"]}\nCI_FLEET_GITHUB_APP_CLIENT_ID={value["client_id"]}\nCI_FLEET_GITHUB_APP_SLUG={slug}\n'); os.chmod(state,0o600) +PY + rm -f "$conversion" + load_metadata +fi + +app_jwt=$temporary/app.jwt +make_app_jwt "$CI_FLEET_GITHUB_APP_ID" "$pem" "$app_jwt" +app_response=$temporary/app.json +write_auth_config "$app_jwt" GET https://api.github.com/app "$app_response" +mapfile -t app_data < <(python3 - "$app_response" "$organization" "$CI_FLEET_GITHUB_APP_SLUG" <<'PY' +import json,sys +v=json.load(open(sys.argv[1])); owner=v.get('owner',{}); expected=sys.argv[3] +if (expected and v.get('slug') != expected) or owner.get('login','').lower() != sys.argv[2].lower() or owner.get('type') != 'Organization': raise SystemExit(1) +permissions=v.get('permissions',{}) +expected_permissions={'contents':'read','metadata':'read','organization_self_hosted_runners':'write'} +if permissions != expected_permissions or v.get('events') not in ([], None) or v.get('public') is not False: raise SystemExit(1) +print(v['slug']); print(v['id']) +PY +) || die 'existing App identity or ownership is unexpected' +((${#app_data[@]} == 2)) || die 'existing App response is incomplete' +CI_FLEET_GITHUB_APP_SLUG=${app_data[0]} +verified_app_id=${app_data[1]} +rm -f "$app_response" +if [[ -n ${handoff_file:-} ]]; then + printf 'https://github.com/apps/%s/installations/new\n' "$CI_FLEET_GITHUB_APP_SLUG" >"$handoff_file" +fi + +installation_id=${CI_FLEET_GITHUB_APP_INSTALLATION_ID:-} +if [[ -z $installation_id ]]; then + note "INSTALLATION_URL https://github.com/apps/$CI_FLEET_GITHUB_APP_SLUG/installations/new" + note 'Install the App for only the listed private repositories, then return here; polling continues automatically.' + deadline=$((SECONDS + timeout)) + while ((SECONDS < deadline)); do + installations=$temporary/installations.json + if write_auth_config "$app_jwt" GET https://api.github.com/app/installations "$installations"; then + installation_id=$(python3 - "$installations" "$organization" <<'PY' +import json,sys +for value in json.load(open(sys.argv[1])): + account=value.get('account',{}) + if account.get('type')=='Organization' and account.get('login','').lower()==sys.argv[2].lower(): + print(value['id']); break +PY +) + fi + rm -f "$installations" + [[ $installation_id =~ ^[0-9]+$ ]] && break + sleep 5 + done + [[ $installation_id =~ ^[0-9]+$ ]] || die 'GitHub App installation was not observed before timeout' +fi + +installation_response=$temporary/installation-token.json +empty_payload=$temporary/empty.json +printf '{}\n' >"$empty_payload" +write_auth_config "$app_jwt" POST "https://api.github.com/app/installations/$installation_id/access_tokens" "$installation_response" "$empty_payload" +installation_token=$temporary/installation.token +python3 - "$installation_response" "$installation_token" <<'PY' +import json,os,sys +from pathlib import Path +value=json.load(open(sys.argv[1])); token=value.get('token') +if not isinstance(token,str) or len(token)<20: raise SystemExit(1) +Path(sys.argv[2]).write_text(token); os.chmod(sys.argv[2],0o600) +PY +rm -f "$installation_response" + +repository_ids=$temporary/repository-ids +: >"$repository_ids" +for repository in "${repositories[@]}"; do + response=$temporary/repository.json + write_auth_config "$installation_token" GET "https://api.github.com/repos/$repository" "$response" + python3 - "$response" "$organization" "$repository" >>"$repository_ids" <<'PY' +import json,sys +v=json.load(open(sys.argv[1])) +if v.get('full_name','').lower()!=sys.argv[3].lower() or v.get('owner',{}).get('login','').lower()!=sys.argv[2].lower() or not v.get('private') or v.get('archived'): raise SystemExit(1) +print(v['id']) +PY + rm -f "$response" +done +sort -n -u -o "$repository_ids" "$repository_ids" + +visible=$temporary/visible.json +write_auth_config "$installation_token" GET 'https://api.github.com/installation/repositories?per_page=100' "$visible" +python3 - "$visible" "${repositories[@]}" <<'PY' || die 'App installation repository access is broader or narrower than requested' +import json,sys +actual=sorted(v['full_name'].lower() for v in json.load(open(sys.argv[1])).get('repositories',[])) +expected=sorted(v.lower() for v in sys.argv[2:]) +if actual != expected: raise SystemExit(1) +PY +rm -f "$visible" + +groups=$temporary/groups.json +write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups?per_page=100" "$groups" +group_id=$(python3 - "$groups" "$runner_group" <<'PY' +import json,sys +for value in json.load(open(sys.argv[1])).get('runner_groups',[]): + if value.get('name')==sys.argv[2]: print(value['id']); break +PY +) +rm -f "$groups" +if [[ -z $group_id ]]; then + [[ $mode != check ]] || die 'runner group is missing' + payload=$temporary/group-create.json + python3 - "$payload" "$runner_group" "$repository_ids" <<'PY' +import json,sys +from pathlib import Path +ids=[int(v) for v in Path(sys.argv[3]).read_text().splitlines()] +Path(sys.argv[1]).write_text(json.dumps({'name':sys.argv[2],'visibility':'selected','selected_repository_ids':ids})) +PY + response=$temporary/group.json + write_auth_config "$installation_token" POST "https://api.github.com/orgs/$organization/actions/runner-groups" "$response" "$payload" + group_id=$(python3 - "$response" <<'PY' +import json,sys +print(json.load(open(sys.argv[1]))['id']) +PY +) + rm -f "$payload" "$response" +fi +[[ $group_id =~ ^[0-9]+$ ]] || die 'runner group ID is invalid' +response=$temporary/group.json +write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id" "$response" +python3 - "$response" "$runner_group" <<'PY' || die 'existing runner group is broader or has unexpected identity; no change made' +import json,sys +v=json.load(open(sys.argv[1])) +if v.get('name')!=sys.argv[2] or v.get('visibility')!='selected' or v.get('default'): raise SystemExit(1) +PY +rm -f "$response" +selected=$temporary/group-repositories.json +write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id/repositories?per_page=100" "$selected" +python3 - "$selected" "$repository_ids" <<'PY' || die 'existing runner group repository access differs; no change made' +import json,sys +from pathlib import Path +actual=sorted(v['id'] for v in json.load(open(sys.argv[1])).get('repositories',[])); expected=sorted(int(v) for v in Path(sys.argv[2]).read_text().splitlines()) +if actual!=expected: raise SystemExit(1) +PY +rm -f "$selected" + +staged=$temporary/host.env +printf '%s\n' "CI_FLEET_GITHUB_APP_CLIENT_ID=$CI_FLEET_GITHUB_APP_CLIENT_ID" "CI_FLEET_GITHUB_APP_INSTALLATION_ID=$installation_id" "CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE=/etc/ci-fleet/secrets/github-app.pem" 'CI_FLEET_RUNNER_TTL=6h' >"$staged" +if [[ $mode == check ]]; then + cmp -s "$staged" "$host_env" || die 'host configuration differs from verified GitHub state' +else + install -m 0600 "$staged" "$host_env" + rm -f "$bootstrap_state" +fi +note "BOOTSTRAP_OK organization=$organization instance=$instance app=$CI_FLEET_GITHUB_APP_SLUG app_id=$verified_app_id installation_id=$installation_id runner_group=$runner_group runner_group_id=$group_id repositories=${#repositories[@]}" +if [[ $mode == check ]]; then note "CREDENTIALS_VERIFIED pem=$pem host_config=$host_env"; else note "CREDENTIALS_WRITTEN pem=$pem host_config=$host_env"; fi +if $run_installer; then + "$script_dir/install-worker-controller.sh" --install --config-repo "$config_repo" --controller "$instance" --ref "$config_ref" +else + note "NEXT sudo $script_dir/install-worker-controller.sh --install --config-repo OWNER/REPO --controller $instance --ref REVIEWED_COMMIT" +fi diff --git a/scripts/fixtures/fake-bootstrap-curl.py b/scripts/fixtures/fake-bootstrap-curl.py new file mode 100755 index 0000000..6a7fed2 --- /dev/null +++ b/scripts/fixtures/fake-bootstrap-curl.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +import json +import os +from pathlib import Path +import sys + +config = Path(sys.argv[sys.argv.index("--config") + 1]).read_text().splitlines() +values = {} +for line in config: + if " = " in line: + key, value = line.split(" = ", 1) + values.setdefault(key, []).append(value.strip().strip('"')) +url = values["url"][0] +endpoint = url.split("?", 1)[0] +method = values.get("request", ["GET"])[0] +output = Path(values["output"][0]) +state = Path(os.environ["FAKE_BOOTSTRAP_STATE"]) +pem = Path(os.environ["FAKE_BOOTSTRAP_PEM_FILE"]).read_text() +response = None +kind = "unknown" +if "/app-manifests/" in url: + kind = "manifest-conversion" + response = {"id": 123, "client_id": "Iv1FixtureClient", "slug": "ci-fleet-example-org-example-ci-01", "pem": pem, + "client_secret": "fixture-client-secret", "webhook_secret": "fixture-webhook-secret"} +elif endpoint.endswith("/app"): + kind = "app" + response = {"id": 123, "slug": "ci-fleet-example-org-example-ci-01", "public": False, "events": [], "owner": {"login": "example-org", "type": "Organization"}, + "permissions": {"contents": "read", "metadata": "read", "organization_self_hosted_runners": "write"}} +elif endpoint.endswith("/app/installations"): + kind = "installations" + response = [{"id": 456, "account": {"login": "example-org", "type": "Organization"}}] +elif endpoint.endswith("/access_tokens"): + kind = "installation-token" + response = {"token": "fixture-installation-token-value"} +elif "/repos/example-org/example-repo" in url: + kind = "repository" + response = {"id": 101, "full_name": "example-org/example-repo", "private": True, "archived": False, + "owner": {"login": "example-org"}} +elif "/installation/repositories" in url: + kind = "installation-repositories" + response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True}]} +elif endpoint.endswith("/actions/runner-groups") and method == "POST": + kind = "runner-group-create" + state.write_text("created\n") + response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False} +elif endpoint.endswith("/actions/runner-groups"): + kind = "runner-groups" + groups = [{"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False}] if state.exists() else [] + response = {"total_count": len(groups), "runner_groups": groups} +elif endpoint.endswith("/actions/runner-groups/789/repositories"): + kind = "runner-group-repositories" + response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True}]} +elif endpoint.endswith("/actions/runner-groups/789"): + kind = "runner-group" + response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False} +else: + raise SystemExit(f"unexpected fake API request: {method} {url}") +with Path(os.environ["FAKE_BOOTSTRAP_LOG"]).open("a") as log: + log.write(f"{method} {kind}\n") +output.write_text(json.dumps(response)) diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh new file mode 100755 index 0000000..90b7847 --- /dev/null +++ b/scripts/test-bootstrap-github.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +bootstrap=$repo_root/scripts/bootstrap-github.sh +callback=$repo_root/scripts/bootstrap-github-callback.py +real_curl=$(command -v curl) +tmp=$(mktemp -d) +processes=() +cleanup() { + local process + for process in "${processes[@]}"; do kill "$process" >/dev/null 2>&1 || true; done + [[ ${KEEP_TEST_TMP:-0} == 1 ]] || rm -rf "$tmp" +} +trap cleanup EXIT +fail() { printf 'FAIL %s\n' "$*" >&2; exit 1; } +port() { python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()'; } +wait_http() { + local url=$1 output=$2 + for _ in {1..100}; do + if "$real_curl" -fsS "$url" -o "$output" 2>/dev/null; then return; fi + sleep 0.05 + done + fail "callback did not become ready: $url" +} + +# The callback rejects mismatched state, never logs, and writes the code once with mode 0600. +callback_port=$(port) +code_file=$tmp/callback-code +handoff_file=$tmp/callback-handoff +python3 "$callback" --bind 127.0.0.1 --port "$callback_port" --organization example-org --state fixture-state \ + --manifest '{"name":"fixture"}' --output "$code_file" --handoff "$handoff_file" --timeout 30 >"$tmp/callback.out" 2>"$tmp/callback.err" & +processes+=("$!") +wait_http "http://127.0.0.1:$callback_port/" "$tmp/callback.html" +status=$("$real_curl" -sS -o "$tmp/wrong.html" -w '%{http_code}' "http://127.0.0.1:$callback_port/callback?code=fixture-code&state=wrong") +[[ $status == 403 && ! -e $code_file ]] || fail 'callback accepted the wrong state' +"$real_curl" -fsS "http://127.0.0.1:$callback_port/callback?code=fixture-code&state=fixture-state" -o "$tmp/correct.html" +printf 'https://github.com/apps/example-fixture/installations/new\n' >"$handoff_file" +"$real_curl" -fsS "http://127.0.0.1:$callback_port/next" -o "$tmp/next.html" +wait "${processes[-1]}" +[[ $(<"$code_file") == fixture-code && $(stat -c %a "$code_file") == 600 ]] || fail 'callback did not protect the conversion code' +[[ ! -s $tmp/callback.out && ! -s $tmp/callback.err ]] || fail 'callback logged request material' +grep -Fq 'https://github.com/apps/example-fixture/installations/new' "$tmp/next.html" || fail 'callback did not provide the installation approval link' + +# Dry-run validates a complete fictional request and performs no write or API call. +dry_root=$tmp/dry-root +CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$dry_root "$bootstrap" --dry-run --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo >"$tmp/dry.out" +grep -Fq 'NO_GITHUB_MUTATION' "$tmp/dry.out" || fail 'dry-run omitted its no-mutation result' +[[ ! -e $dry_root ]] || fail 'dry-run wrote host state' +if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group Default --allow-repository example-org/example-repo >/dev/null 2>&1; then + fail 'default runner group was accepted' +fi +if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group example-ci-experimental \ + --allow-repository example-org/example-repo --bind 0.0.0.0 --callback-host 0.0.0.0 >/dev/null 2>&1; then + fail 'unspecified callback bind was accepted' +fi +unsafe_root=$tmp/unsafe-root +mkdir -p "$unsafe_root/etc" "$tmp/redirected" +ln -s "$tmp/redirected" "$unsafe_root/etc/ci-fleet" +if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$unsafe_root "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then + fail 'symlinked protected directory was accepted' +fi + +# Mock the GitHub API while exercising the complete local callback, conversion, persistence, +# exact private-repository/group checks, idempotent check, and redaction paths. +fake_bin=$tmp/bin +mkdir "$fake_bin" +cp "$repo_root/scripts/fixtures/fake-bootstrap-curl.py" "$fake_bin/curl" +chmod 0755 "$fake_bin/curl" +openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "$tmp/fixture.pem" >/dev/null 2>&1 +export FAKE_BOOTSTRAP_STATE=$tmp/group-created FAKE_BOOTSTRAP_PEM_FILE=$tmp/fixture.pem FAKE_BOOTSTRAP_LOG=$tmp/api.log +live_root=$tmp/live-root +live_port=$(port) +PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo --port "$live_port" --timeout 30 >"$tmp/live.out" 2>"$tmp/live.err" & +live_pid=$! +processes+=("$live_pid") +wait_http "http://127.0.0.1:$live_port/" "$tmp/live.html" +state=$(python3 - "$tmp/live.html" <<'PY' +import html,re,sys,urllib.parse +body=html.unescape(open(sys.argv[1]).read()) +match=re.search(r"settings/apps/new\?state=([^'&]+)",body) +if not match: raise SystemExit(1) +print(urllib.parse.unquote(match.group(1))) +PY +) +"$real_curl" -fsS --get --data-urlencode code=fixture-conversion-code --data-urlencode "state=$state" \ + "http://127.0.0.1:$live_port/callback" -o "$tmp/live-callback.html" +wait "$live_pid" +grep -Fq 'BOOTSTRAP_OK organization=example-org instance=example-ci-01 app=ci-fleet-example-org-example-ci-01 app_id=123 installation_id=456 runner_group=example-ci-experimental runner_group_id=789 repositories=1' "$tmp/live.out" || fail 'live bootstrap did not complete' +host_env=$live_root/etc/ci-fleet/host.env +pem=$live_root/etc/ci-fleet/secrets/github-app.pem +[[ -f $host_env && -f $pem && $(stat -c %a "$host_env") == 600 && $(stat -c %a "$pem") == 600 ]] || fail 'credentials were not stored with mode 0600' +grep -Fxq 'CI_FLEET_GITHUB_APP_CLIENT_ID=Iv1FixtureClient' "$host_env" || fail 'client ID was not recorded' +grep -Fxq 'CI_FLEET_GITHUB_APP_INSTALLATION_ID=456' "$host_env" || fail 'installation ID was not recorded' +[[ ! -e $live_root/etc/ci-fleet/bootstrap-app.env ]] || fail 'partial bootstrap metadata survived success' +if grep -R -F -e fixture-conversion-code -e fixture-client-secret -e fixture-webhook-secret -e fixture-installation-token-value "$tmp/live.out" "$tmp/live.err" "$tmp/api.log"; then + fail 'bootstrap output exposed sensitive fixture material' +fi +grep -Fxq 'POST runner-group-create' "$tmp/api.log" || fail 'runner group was not created in the mocked live flow' + +before=$(sha256sum "$host_env" "$pem") +PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo >"$tmp/check.out" 2>"$tmp/check.err" +[[ $(sha256sum "$host_env" "$pem") == "$before" ]] || fail 'check mode changed credentials' +grep -Fq 'CREDENTIALS_VERIFIED' "$tmp/check.out" || fail 'check mode did not verify persisted credentials' +[[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == 1 ]] || fail 'check mode mutated the runner group' + +printf 'BOOTSTRAP_TESTS_OK\n' diff --git a/scripts/validate.sh b/scripts/validate.sh index 9660aaa..626841d 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -9,6 +9,8 @@ python3 -m py_compile \ .github/actions/plan/plan.py \ .github/actions/plan/test_plan.py \ scripts/desired_state.py \ + scripts/bootstrap-github-callback.py \ + scripts/fixtures/fake-bootstrap-curl.py \ scripts/health.py \ scripts/status_auth.py \ scripts/status_receiver.py \ @@ -22,6 +24,7 @@ python3 scripts/test_desired_state.py python3 scripts/test_health.py python3 scripts/test_status_receiver.py python3 scripts/test_quickstart.py +scripts/test-bootstrap-github.sh python3 -m json.tool schemas/status-report-v1.json >/dev/null python3 scripts/desired_state.py validate-engine-capabilities --manifest engine-capabilities.json --require-status-reporting-config --require-status-reporting >/dev/null python3 .github/actions/plan/plan.py --plan examples/project/scripts/ci/plan.json --group fast >/dev/null From 89d79ace17403844ebd347dde7a89d7183bcdddc Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sat, 15 Aug 2026 23:52:15 -0500 Subject: [PATCH 02/10] fix: make GitHub bootstrap recoverable and scoped --- docs/GITHUB-APP-SETUP.md | 10 +-- scripts/bootstrap-github.sh | 100 ++++++++++++++++-------- scripts/fixtures/fake-bootstrap-curl.py | 14 +++- scripts/test-bootstrap-github.sh | 29 ++++++- 4 files changed, 105 insertions(+), 48 deletions(-) diff --git a/docs/GITHUB-APP-SETUP.md b/docs/GITHUB-APP-SETUP.md index 19ada63..fdd23b1 100644 --- a/docs/GITHUB-APP-SETUP.md +++ b/docs/GITHUB-APP-SETUP.md @@ -18,9 +18,9 @@ organization, app name, and configuration repository. ## Bootstrap on the target host -Run the reviewed checkout on the target Linux Docker host. The callback defaults -to loopback. For phone-first use, explicitly choose one private address that the -phone can reach; the script rejects public callback addresses. +Run the reviewed checkout on the target Linux Docker host. The plaintext callback +is loopback-only; non-loopback callbacks are rejected because the one-time +manifest conversion code requires transport confidentiality. ```bash sudo ./scripts/bootstrap-github.sh \ @@ -30,10 +30,6 @@ sudo ./scripts/bootstrap-github.sh \ --allow-repository example-org/example-repo ``` -For phone access, append `--bind PRIVATE_IP --callback-host PRIVATE_IP` using the -same explicitly selected host-local private address; do not publish that address -in Git, logs, or support messages. - All names above are fictional. The script prints one non-secret local `REGISTRATION_URL`. Open it, press the single registration button, install the new App for **only** the requested private repositories, and return to the diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh index 756be4e..4003d09 100755 --- a/scripts/bootstrap-github.sh +++ b/scripts/bootstrap-github.sh @@ -36,7 +36,7 @@ while (($#)); do esac done [[ $organization =~ ^[A-Za-z0-9][A-Za-z0-9-]{0,38}$ ]] || die 'invalid organization' -[[ $instance =~ ^[a-z0-9][a-z0-9.-]{0,62}$ ]] || die 'invalid instance identity' +[[ $instance =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] || die 'invalid instance identity' [[ $runner_group =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$ && ${runner_group,,} != default ]] || die 'runner group must be explicit and non-default' ((${#repositories[@]} > 0)) || die 'at least one --allow-repository is required' [[ $port =~ ^[0-9]+$ && $port -ge 1 && $port -le 65535 ]] || die 'invalid callback port' @@ -61,6 +61,7 @@ secret_dir=$etc_dir/secrets pem=$secret_dir/github-app.pem host_env=$etc_dir/host.env bootstrap_state=$etc_dir/bootstrap-app.env +bootstrap_recovery=$etc_dir/bootstrap-recovery.json temporary=$(mktemp -d) callback_pid= expected_uid=0 @@ -75,12 +76,12 @@ cleanup() { trap cleanup EXIT INT TERM HUP if [[ $mode != dry-run && ${CI_FLEET_TESTING:-0} != 1 && ${EUID:-$(id -u)} -ne 0 ]]; then die 'run live/check bootstrap as root'; fi -python3 - "$bind" "$callback_host" <<'PY' || die 'callback bind/host must be loopback or private IP addresses' +python3 - "$bind" "$callback_host" <<'PY' || die 'callback bind/host must be the same loopback IP address' import ipaddress, sys addresses=[ipaddress.ip_address(value) for value in sys.argv[1:]] if addresses[0] != addresses[1]: raise SystemExit(1) address=addresses[0] -if address.is_unspecified or not (address.is_loopback or address.is_private): raise SystemExit(1) +if not address.is_loopback: raise SystemExit(1) PY write_auth_config() { @@ -119,19 +120,40 @@ load_metadata() { # shellcheck disable=SC1090 . "$bootstrap_state" [[ ${CI_FLEET_BOOTSTRAP_ORGANIZATION:-} == "$organization" && ${CI_FLEET_BOOTSTRAP_INSTANCE:-} == "$instance" ]] || die 'existing bootstrap state belongs to another identity' - elif [[ -f $host_env && ! -L $host_env && $(stat -c %a "$host_env") == 600 && $(stat -c %u "$host_env") == "$expected_uid" ]]; then - # shellcheck disable=SC1090 - . "$host_env" - CI_FLEET_GITHUB_APP_ID=${CI_FLEET_GITHUB_APP_CLIENT_ID:-} - CI_FLEET_GITHUB_APP_SLUG= + if [[ -e $host_env || -L $host_env ]]; then + [[ -f $host_env && ! -L $host_env && $(stat -c %a "$host_env") == 600 && $(stat -c %u "$host_env") == "$expected_uid" ]] || die 'existing host configuration is invalid' + # shellcheck disable=SC1090 + . "$host_env" + fi else return 1 fi - [[ ${CI_FLEET_GITHUB_APP_ID:-} =~ ^([0-9]+|(Iv1\.)?[A-Za-z0-9]+)$ && ${CI_FLEET_GITHUB_APP_CLIENT_ID:-} =~ ^(Iv1\.)?[A-Za-z0-9]+$ && ( -z ${CI_FLEET_GITHUB_APP_SLUG:-} || ${CI_FLEET_GITHUB_APP_SLUG:-} =~ ^[a-z0-9-]+$ ) ]] || die 'existing bootstrap state is invalid' + [[ ${CI_FLEET_GITHUB_APP_ID:-} =~ ^([0-9]+|(Iv1\.)?[A-Za-z0-9]+)$ && ${CI_FLEET_GITHUB_APP_CLIENT_ID:-} =~ ^(Iv1\.)?[A-Za-z0-9]+$ && ${CI_FLEET_GITHUB_APP_SLUG:-} =~ ^[a-z0-9-]+$ ]] || die 'existing bootstrap state is invalid' [[ ${CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE:-/etc/ci-fleet/secrets/github-app.pem} == /etc/ci-fleet/secrets/github-app.pem ]] || die 'existing bootstrap PEM path is unsupported' [[ -f $pem && ! -L $pem && $(stat -c %a "$pem") == 600 && $(stat -c %u "$pem") == "$expected_uid" ]] || die 'existing bootstrap PEM is invalid' } +recover_conversion() { + [[ -f $bootstrap_recovery && ! -L $bootstrap_recovery && $(stat -c %a "$bootstrap_recovery") == 600 && $(stat -c %u "$bootstrap_recovery") == "$expected_uid" ]] || die 'bootstrap recovery record is invalid' + local staged_pem=$temporary/recovered.pem staged_state=$temporary/recovered.env + python3 - "$bootstrap_recovery" "$staged_pem" "$staged_state" "$organization" "$instance" <<'PY' || die 'bootstrap recovery record is malformed' +import json, os, re, sys +from pathlib import Path +source, pem, state = map(Path, sys.argv[1:4]); organization, instance = sys.argv[4:] +value=json.loads(source.read_text()) +if not isinstance(value.get('pem'),str) or 'PRIVATE KEY' not in value['pem']: raise SystemExit(1) +if not isinstance(value.get('id'),int) or not re.fullmatch(r'(Iv1\.)?[A-Za-z0-9]+',str(value.get('client_id',''))): raise SystemExit(1) +slug=str(value.get('slug','')) +if slug != f'ci-fleet-{organization}-{instance}' or not re.fullmatch(r'[a-z0-9-]+',slug): raise SystemExit(1) +pem.write_text(value['pem']); os.chmod(pem,0o600) +state.write_text(f'CI_FLEET_BOOTSTRAP_ORGANIZATION={organization}\nCI_FLEET_BOOTSTRAP_INSTANCE={instance}\nCI_FLEET_GITHUB_APP_ID={value["id"]}\nCI_FLEET_GITHUB_APP_CLIENT_ID={value["client_id"]}\nCI_FLEET_GITHUB_APP_SLUG={slug}\n'); os.chmod(state,0o600) +PY + install -m 0600 "$staged_pem" "$pem" + install -m 0600 "$staged_state" "$bootstrap_state" + load_metadata + rm -f "$bootstrap_recovery" +} + if [[ $mode == dry-run ]]; then note "DRY_RUN organization=$organization instance=$instance runner_group=$runner_group repositories=${#repositories[@]}" note "CALLBACK_URL http://$callback_host:$port/" @@ -146,6 +168,10 @@ for directory in "$etc_dir" "$secret_dir"; do install -d -m 0700 "$directory" fi done +if [[ -e $bootstrap_recovery || -L $bootstrap_recovery ]]; then + [[ $mode != check ]] || die 'bootstrap credential recovery is pending; rerun live bootstrap first' + recover_conversion +fi if ! load_metadata; then [[ $mode != check ]] || die 'bootstrap state is missing; run live bootstrap first' [[ ! -e $pem && ! -L $pem && ! -e $host_env && ! -L $host_env ]] || die 'existing credentials lack matching bootstrap identity; no replacement made' @@ -174,9 +200,8 @@ PY sleep 1 done [[ -f $code_file ]] || die 'registration callback timed out' - conversion=$temporary/conversion.json conversion_config=$temporary/conversion.conf - python3 - "$code_file" "$conversion" "$conversion_config" <<'PY' + python3 - "$code_file" "$bootstrap_recovery" "$conversion_config" <<'PY' from pathlib import Path import sys code_file, output, config = map(Path, sys.argv[1:]) @@ -187,22 +212,11 @@ config.write_text('\n'.join(['silent','show-error','fail-with-body','request = " 'header = "Accept: application/vnd.github+json"','header = "X-GitHub-Api-Version: 2022-11-28"']) + '\n') PY chmod 600 "$conversion_config" - curl --config "$conversion_config" + if ! curl --config "$conversion_config"; then rm -f "$bootstrap_recovery"; die 'manifest conversion failed'; fi + chmod 600 "$bootstrap_recovery" rm -f "$code_file" "$conversion_config" - python3 - "$conversion" "$pem" "$bootstrap_state" "$organization" "$instance" <<'PY' -import json, os, re, sys -from pathlib import Path -source, pem, state = map(Path, sys.argv[1:4]); organization, instance = sys.argv[4:] -value=json.loads(source.read_text()) -if not isinstance(value.get('pem'),str) or 'PRIVATE KEY' not in value['pem']: raise SystemExit(1) -if not isinstance(value.get('id'),int) or not re.fullmatch(r'(Iv1\.)?[A-Za-z0-9]+',str(value.get('client_id',''))): raise SystemExit(1) -slug=str(value.get('slug','')) -if not re.fullmatch(r'[a-z0-9-]+',slug): raise SystemExit(1) -pem.write_text(value['pem']); os.chmod(pem,0o600) -state.write_text(f'CI_FLEET_BOOTSTRAP_ORGANIZATION={organization}\nCI_FLEET_BOOTSTRAP_INSTANCE={instance}\nCI_FLEET_GITHUB_APP_ID={value["id"]}\nCI_FLEET_GITHUB_APP_CLIENT_ID={value["client_id"]}\nCI_FLEET_GITHUB_APP_SLUG={slug}\n'); os.chmod(state,0o600) -PY - rm -f "$conversion" - load_metadata + if [[ ${CI_FLEET_TEST_FAIL_AFTER_CONVERSION:-0} == 1 && ${CI_FLEET_TESTING:-0} == 1 ]]; then die 'injected failure after manifest conversion'; fi + recover_conversion fi app_jwt=$temporary/app.jwt @@ -234,12 +248,13 @@ if [[ -z $installation_id ]]; then deadline=$((SECONDS + timeout)) while ((SECONDS < deadline)); do installations=$temporary/installations.json + make_app_jwt "$CI_FLEET_GITHUB_APP_ID" "$pem" "$app_jwt" if write_auth_config "$app_jwt" GET https://api.github.com/app/installations "$installations"; then installation_id=$(python3 - "$installations" "$organization" <<'PY' import json,sys for value in json.load(open(sys.argv[1])): account=value.get('account',{}) - if account.get('type')=='Organization' and account.get('login','').lower()==sys.argv[2].lower(): + if value.get('repository_selection')=='selected' and account.get('type')=='Organization' and account.get('login','').lower()==sys.argv[2].lower(): print(value['id']); break PY ) @@ -251,9 +266,23 @@ PY [[ $installation_id =~ ^[0-9]+$ ]] || die 'GitHub App installation was not observed before timeout' fi +make_app_jwt "$CI_FLEET_GITHUB_APP_ID" "$pem" "$app_jwt" +installations=$temporary/installations.json +write_auth_config "$app_jwt" GET https://api.github.com/app/installations "$installations" +python3 - "$installations" "$organization" "$installation_id" <<'PY' || die 'App installation identity or repository selection is unexpected' +import json,sys +matches=[] +for value in json.load(open(sys.argv[1])): + account=value.get('account',{}) + if str(value.get('id'))==sys.argv[3] and account.get('type')=='Organization' and account.get('login','').lower()==sys.argv[2].lower() and value.get('repository_selection')=='selected': matches.append(value) +if len(matches)!=1: raise SystemExit(1) +PY +rm -f "$installations" + installation_response=$temporary/installation-token.json empty_payload=$temporary/empty.json printf '{}\n' >"$empty_payload" +make_app_jwt "$CI_FLEET_GITHUB_APP_ID" "$pem" "$app_jwt" write_auth_config "$app_jwt" POST "https://api.github.com/app/installations/$installation_id/access_tokens" "$installation_response" "$empty_payload" installation_token=$temporary/installation.token python3 - "$installation_response" "$installation_token" <<'PY' @@ -284,7 +313,9 @@ visible=$temporary/visible.json write_auth_config "$installation_token" GET 'https://api.github.com/installation/repositories?per_page=100' "$visible" python3 - "$visible" "${repositories[@]}" <<'PY' || die 'App installation repository access is broader or narrower than requested' import json,sys -actual=sorted(v['full_name'].lower() for v in json.load(open(sys.argv[1])).get('repositories',[])) +value=json.load(open(sys.argv[1])); repositories=value.get('repositories',[]) +if value.get('total_count') != len(repositories): raise SystemExit(1) +actual=sorted(v['full_name'].lower() for v in repositories) expected=sorted(v.lower() for v in sys.argv[2:]) if actual != expected: raise SystemExit(1) PY @@ -294,7 +325,9 @@ groups=$temporary/groups.json write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups?per_page=100" "$groups" group_id=$(python3 - "$groups" "$runner_group" <<'PY' import json,sys -for value in json.load(open(sys.argv[1])).get('runner_groups',[]): +response=json.load(open(sys.argv[1])); groups=response.get('runner_groups',[]) +if response.get('total_count') != len(groups): raise SystemExit(1) +for value in groups: if value.get('name')==sys.argv[2]: print(value['id']); break PY ) @@ -306,7 +339,7 @@ if [[ -z $group_id ]]; then import json,sys from pathlib import Path ids=[int(v) for v in Path(sys.argv[3]).read_text().splitlines()] -Path(sys.argv[1]).write_text(json.dumps({'name':sys.argv[2],'visibility':'selected','selected_repository_ids':ids})) +Path(sys.argv[1]).write_text(json.dumps({'name':sys.argv[2],'visibility':'selected','allows_public_repositories':False,'selected_repository_ids':ids})) PY response=$temporary/group.json write_auth_config "$installation_token" POST "https://api.github.com/orgs/$organization/actions/runner-groups" "$response" "$payload" @@ -323,7 +356,7 @@ write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organi python3 - "$response" "$runner_group" <<'PY' || die 'existing runner group is broader or has unexpected identity; no change made' import json,sys v=json.load(open(sys.argv[1])) -if v.get('name')!=sys.argv[2] or v.get('visibility')!='selected' or v.get('default'): raise SystemExit(1) +if v.get('name')!=sys.argv[2] or v.get('visibility')!='selected' or v.get('default') or v.get('allows_public_repositories') is not False: raise SystemExit(1) PY rm -f "$response" selected=$temporary/group-repositories.json @@ -331,7 +364,9 @@ write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organi python3 - "$selected" "$repository_ids" <<'PY' || die 'existing runner group repository access differs; no change made' import json,sys from pathlib import Path -actual=sorted(v['id'] for v in json.load(open(sys.argv[1])).get('repositories',[])); expected=sorted(int(v) for v in Path(sys.argv[2]).read_text().splitlines()) +response=json.load(open(sys.argv[1])); repositories=response.get('repositories',[]) +if response.get('total_count') != len(repositories): raise SystemExit(1) +actual=sorted(v['id'] for v in repositories); expected=sorted(int(v) for v in Path(sys.argv[2]).read_text().splitlines()) if actual!=expected: raise SystemExit(1) PY rm -f "$selected" @@ -342,7 +377,6 @@ if [[ $mode == check ]]; then cmp -s "$staged" "$host_env" || die 'host configuration differs from verified GitHub state' else install -m 0600 "$staged" "$host_env" - rm -f "$bootstrap_state" fi note "BOOTSTRAP_OK organization=$organization instance=$instance app=$CI_FLEET_GITHUB_APP_SLUG app_id=$verified_app_id installation_id=$installation_id runner_group=$runner_group runner_group_id=$group_id repositories=${#repositories[@]}" if [[ $mode == check ]]; then note "CREDENTIALS_VERIFIED pem=$pem host_config=$host_env"; else note "CREDENTIALS_WRITTEN pem=$pem host_config=$host_env"; fi diff --git a/scripts/fixtures/fake-bootstrap-curl.py b/scripts/fixtures/fake-bootstrap-curl.py index 6a7fed2..eb4755b 100755 --- a/scripts/fixtures/fake-bootstrap-curl.py +++ b/scripts/fixtures/fake-bootstrap-curl.py @@ -28,7 +28,7 @@ "permissions": {"contents": "read", "metadata": "read", "organization_self_hosted_runners": "write"}} elif endpoint.endswith("/app/installations"): kind = "installations" - response = [{"id": 456, "account": {"login": "example-org", "type": "Organization"}}] + response = [{"id": 456, "repository_selection": "selected", "account": {"login": "example-org", "type": "Organization"}}] elif endpoint.endswith("/access_tokens"): kind = "installation-token" response = {"token": "fixture-installation-token-value"} @@ -42,19 +42,25 @@ elif endpoint.endswith("/actions/runner-groups") and method == "POST": kind = "runner-group-create" state.write_text("created\n") - response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False} + response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False} elif endpoint.endswith("/actions/runner-groups"): kind = "runner-groups" - groups = [{"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False}] if state.exists() else [] + groups = [{"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False}] if state.exists() else [] response = {"total_count": len(groups), "runner_groups": groups} elif endpoint.endswith("/actions/runner-groups/789/repositories"): kind = "runner-group-repositories" response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True}]} elif endpoint.endswith("/actions/runner-groups/789"): kind = "runner-group" - response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False} + response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False} else: raise SystemExit(f"unexpected fake API request: {method} {url}") +if kind == "installations" and os.environ.get("FAKE_BOOTSTRAP_ALL_REPOSITORIES") == "1": + response[0]["repository_selection"] = "all" +if kind in {"runner-group", "runner-groups"} and os.environ.get("FAKE_BOOTSTRAP_PUBLIC_GROUP") == "1": + (response["runner_groups"][0] if kind == "runner-groups" else response)["allows_public_repositories"] = True +if kind == os.environ.get("FAKE_BOOTSTRAP_TRUNCATE_KIND"): + response["total_count"] += 1 with Path(os.environ["FAKE_BOOTSTRAP_LOG"]).open("a") as log: log.write(f"{method} {kind}\n") output.write_text(json.dumps(response)) diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh index 90b7847..eafcb17 100755 --- a/scripts/test-bootstrap-github.sh +++ b/scripts/test-bootstrap-github.sh @@ -51,6 +51,13 @@ grep -Fq 'NO_GITHUB_MUTATION' "$tmp/dry.out" || fail 'dry-run omitted its no-mut if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group Default --allow-repository example-org/example-repo >/dev/null 2>&1; then fail 'default runner group was accepted' fi +if "$bootstrap" --dry-run --organization example-org --instance example.ci --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then + fail 'instance outside the installer contract was accepted' +fi +if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group example-ci-experimental \ + --allow-repository example-org/example-repo --bind 10.0.0.1 --callback-host 10.0.0.1 >/dev/null 2>&1; then + fail 'plaintext non-loopback callback was accepted' +fi if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group example-ci-experimental \ --allow-repository example-org/example-repo --bind 0.0.0.0 --callback-host 0.0.0.0 >/dev/null 2>&1; then fail 'unspecified callback bind was accepted' @@ -73,8 +80,8 @@ openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "$tmp/fixture. export FAKE_BOOTSTRAP_STATE=$tmp/group-created FAKE_BOOTSTRAP_PEM_FILE=$tmp/fixture.pem FAKE_BOOTSTRAP_LOG=$tmp/api.log live_root=$tmp/live-root live_port=$(port) -PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo --port "$live_port" --timeout 30 >"$tmp/live.out" 2>"$tmp/live.err" & +PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_TEST_FAIL_AFTER_CONVERSION=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo --port "$live_port" --timeout 30 >"$tmp/fault.out" 2>"$tmp/fault.err" & live_pid=$! processes+=("$live_pid") wait_http "http://127.0.0.1:$live_port/" "$tmp/live.html" @@ -88,14 +95,17 @@ PY ) "$real_curl" -fsS --get --data-urlencode code=fixture-conversion-code --data-urlencode "state=$state" \ "http://127.0.0.1:$live_port/callback" -o "$tmp/live-callback.html" -wait "$live_pid" +if wait "$live_pid"; then fail 'injected post-conversion failure succeeded'; fi +[[ -f $live_root/etc/ci-fleet/bootstrap-recovery.json && ! -e $live_root/etc/ci-fleet/secrets/github-app.pem ]] || fail 'converted credentials were not left recoverable' +PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo --port "$live_port" --timeout 30 >"$tmp/live.out" 2>"$tmp/live.err" grep -Fq 'BOOTSTRAP_OK organization=example-org instance=example-ci-01 app=ci-fleet-example-org-example-ci-01 app_id=123 installation_id=456 runner_group=example-ci-experimental runner_group_id=789 repositories=1' "$tmp/live.out" || fail 'live bootstrap did not complete' host_env=$live_root/etc/ci-fleet/host.env pem=$live_root/etc/ci-fleet/secrets/github-app.pem [[ -f $host_env && -f $pem && $(stat -c %a "$host_env") == 600 && $(stat -c %a "$pem") == 600 ]] || fail 'credentials were not stored with mode 0600' grep -Fxq 'CI_FLEET_GITHUB_APP_CLIENT_ID=Iv1FixtureClient' "$host_env" || fail 'client ID was not recorded' grep -Fxq 'CI_FLEET_GITHUB_APP_INSTALLATION_ID=456' "$host_env" || fail 'installation ID was not recorded' -[[ ! -e $live_root/etc/ci-fleet/bootstrap-app.env ]] || fail 'partial bootstrap metadata survived success' +[[ -f $live_root/etc/ci-fleet/bootstrap-app.env && ! -e $live_root/etc/ci-fleet/bootstrap-recovery.json ]] || fail 'durable identity or recovery cleanup is incorrect' if grep -R -F -e fixture-conversion-code -e fixture-client-secret -e fixture-webhook-secret -e fixture-installation-token-value "$tmp/live.out" "$tmp/live.err" "$tmp/api.log"; then fail 'bootstrap output exposed sensitive fixture material' fi @@ -108,4 +118,15 @@ PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$boot grep -Fq 'CREDENTIALS_VERIFIED' "$tmp/check.out" || fail 'check mode did not verify persisted credentials' [[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == 1 ]] || fail 'check mode mutated the runner group' +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --check --organization example-org --instance other-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then fail 'persisted App was rebound to another instance'; fi +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_ALL_REPOSITORIES=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then fail 'all-repository installation was accepted'; fi +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_PUBLIC_GROUP=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then fail 'public-repository runner group was accepted'; fi +for kind in installation-repositories runner-groups runner-group-repositories; do + if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_TRUNCATE_KIND=$kind "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then fail "truncated $kind page was accepted"; fi +done + printf 'BOOTSTRAP_TESTS_OK\n' From 831844a2aad604928f30d6d20d52089d0ba98fca Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sun, 16 Aug 2026 00:46:15 -0500 Subject: [PATCH 03/10] fix: separate App and runner repository scopes --- docs/GITHUB-APP-SETUP.md | 29 ++++++-- scripts/bootstrap-github.sh | 88 ++++++++++++++----------- scripts/fixtures/fake-bootstrap-curl.py | 16 +++-- scripts/test-bootstrap-github.sh | 30 +++++---- 4 files changed, 105 insertions(+), 58 deletions(-) diff --git a/docs/GITHUB-APP-SETUP.md b/docs/GITHUB-APP-SETUP.md index fdd23b1..06325fb 100644 --- a/docs/GITHUB-APP-SETUP.md +++ b/docs/GITHUB-APP-SETUP.md @@ -26,17 +26,29 @@ manifest conversion code requires transport confidentiality. sudo ./scripts/bootstrap-github.sh \ --organization example-org \ --instance example-ci-01 \ + --config-repository example-org/config \ --runner-group example-ci-experimental \ - --allow-repository example-org/example-repo + --allow-repository example-org/example-repo=123456 ``` All names above are fictional. The script prints one non-secret local `REGISTRATION_URL`. Open it, press the single registration button, install the -new App for **only** the requested private repositories, and return to the +new App for **only** the private configuration repository, and return to the terminal. The target host receives and exchanges the temporary code itself. Neither the code nor any credential is copied through a phone, clipboard, chat, email, issue, or second computer. +For a headless remote host, establish an authenticated SSH local forward from +the management workstation first: `ssh -L 8765:127.0.0.1:8765 HOST`. Keep that +session open, run bootstrap on `HOST`, and open `http://127.0.0.1:8765/` on the +workstation. Close the SSH session immediately after bootstrap. Do not expose +the callback listener or forward on a shared workstation. + +Each project argument includes its numeric repository ID, obtained and reviewed +by the organization owner. The App itself receives access only to the separate +configuration repository; the project name/ID pairs scope runner-group routing +without granting the controller App project source access. + The bootstrap: - requests only `contents: read`, metadata read, and organization self-hosted @@ -49,16 +61,23 @@ The bootstrap: default/broad runner-group access; - creates a missing selected-repository runner group, but never changes an existing group whose identity or access differs; -- destroys the conversion code, conversion response, JWTs, installation token, - callback state, and temporary curl configurations on every exit. +- destroys the conversion code, JWTs, installation token, callback state, and + temporary curl configurations on exit. After a successful conversion, a + protected mode-`0600` `bootstrap-recovery.json` (or atomic-publication + `bootstrap-recovery.pending`) is deliberately retained until the PEM and + identity record are installed. Rerun the identical live command to recover. + If bootstrap is abandoned, an organization owner must first revoke/delete the + exact newly created App, then explicitly remove those recovery files; ordinary + cleanup never discards potentially unique credentials. Inspect a request without local writes or GitHub calls: ```bash sudo ./scripts/bootstrap-github.sh --dry-run \ --organization example-org --instance example-ci-01 \ + --config-repository example-org/config \ --runner-group example-ci-experimental \ - --allow-repository example-org/example-repo + --allow-repository example-org/example-repo=123456 ``` After success, rerun the same command with `--check` to verify App ownership, diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh index 4003d09..34ec11d 100755 --- a/scripts/bootstrap-github.sh +++ b/scripts/bootstrap-github.sh @@ -5,7 +5,8 @@ umask 077 usage() { cat <<'EOF' Usage: bootstrap-github.sh [--dry-run|--check] --organization ORG --instance ID - --runner-group GROUP --allow-repository ORG/REPO [--allow-repository ORG/REPO ...] + --config-repository ORG/PRIVATE_CONFIG_REPO --runner-group GROUP + --allow-repository ORG/PROJECT=NUMERIC_ID [--allow-repository ORG/PROJECT=NUMERIC_ID ...] [--bind ADDRESS --callback-host HOST --port PORT --timeout SECONDS] [--install --config-repo OWNER/REPO_OR_PATH --config-ref COMMIT] EOF @@ -13,7 +14,7 @@ EOF die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } note() { printf '%s\n' "$*"; } -mode=live; organization=; instance=; runner_group=; bind=127.0.0.1; callback_host=127.0.0.1 +mode=live; organization=; instance=; runner_group=; config_repository=; bind=127.0.0.1; callback_host=127.0.0.1 port=8765; timeout=600; run_installer=false; config_repo=; config_ref= repositories=() while (($#)); do @@ -23,6 +24,7 @@ while (($#)); do --organization) (($# >= 2)) || die '--organization requires a value'; organization=$2; shift 2 ;; --instance) (($# >= 2)) || die '--instance requires a value'; instance=$2; shift 2 ;; --runner-group) (($# >= 2)) || die '--runner-group requires a value'; runner_group=$2; shift 2 ;; + --config-repository) (($# >= 2)) || die '--config-repository requires a value'; config_repository=$2; shift 2 ;; --allow-repository) (($# >= 2)) || die '--allow-repository requires a value'; repositories+=("$2"); shift 2 ;; --bind) (($# >= 2)) || die '--bind requires a value'; bind=$2; shift 2 ;; --callback-host) (($# >= 2)) || die '--callback-host requires a value'; callback_host=$2; shift 2 ;; @@ -35,17 +37,22 @@ while (($#)); do *) usage; die "unknown argument: $1" ;; esac done -[[ $organization =~ ^[A-Za-z0-9][A-Za-z0-9-]{0,38}$ ]] || die 'invalid organization' +[[ $organization =~ ^[a-z0-9][a-z0-9-]{0,38}$ ]] || die 'organization must be lowercase' [[ $instance =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] || die 'invalid instance identity' -[[ $runner_group =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$ && ${runner_group,,} != default ]] || die 'runner group must be explicit and non-default' +[[ $runner_group =~ ^[a-z0-9][a-z0-9-]{0,62}$ && $runner_group != default ]] || die 'runner group must be a lowercase schema-v3 slug and non-default' +[[ $config_repository =~ ^${organization}/[A-Za-z0-9_.-]+$ ]] || die "configuration repository must belong to $organization" +app_name=ci-fleet-$organization-$instance +((${#app_name} <= 34)) || die 'generated GitHub App name exceeds 34 characters' ((${#repositories[@]} > 0)) || die 'at least one --allow-repository is required' [[ $port =~ ^[0-9]+$ && $port -ge 1 && $port -le 65535 ]] || die 'invalid callback port' [[ $timeout =~ ^[0-9]+$ && $timeout -ge 30 && $timeout -le 1800 ]] || die 'timeout must be 30-1800 seconds' for repository in "${repositories[@]}"; do - [[ $repository =~ ^${organization}/[A-Za-z0-9_.-]+$ ]] || die "repository must belong to $organization: $repository" + [[ $repository =~ ^${organization}/[A-Za-z0-9_.-]+=([1-9][0-9]*)$ ]] || die "project repository must use ORG/REPO=NUMERIC_ID: $repository" done mapfile -t repositories < <(printf '%s\n' "${repositories[@]}" | LC_ALL=C sort -u) ((${#repositories[@]} > 0)) || die 'repository allowlist is empty' +repository_names=(); repository_id_values=() +for repository in "${repositories[@]}"; do repository_names+=("${repository%=*}"); repository_id_values+=("${repository##*=}"); done if $run_installer; then [[ $mode == live ]] || die '--install is available only in live mode' [[ -n $config_repo && $config_ref =~ ^[0-9a-f]{40}$ ]] || die '--install requires --config-repo and a 40-character --config-ref' @@ -62,6 +69,7 @@ pem=$secret_dir/github-app.pem host_env=$etc_dir/host.env bootstrap_state=$etc_dir/bootstrap-app.env bootstrap_recovery=$etc_dir/bootstrap-recovery.json +bootstrap_pending=$etc_dir/bootstrap-recovery.pending temporary=$(mktemp -d) callback_pid= expected_uid=0 @@ -76,13 +84,7 @@ cleanup() { trap cleanup EXIT INT TERM HUP if [[ $mode != dry-run && ${CI_FLEET_TESTING:-0} != 1 && ${EUID:-$(id -u)} -ne 0 ]]; then die 'run live/check bootstrap as root'; fi -python3 - "$bind" "$callback_host" <<'PY' || die 'callback bind/host must be the same loopback IP address' -import ipaddress, sys -addresses=[ipaddress.ip_address(value) for value in sys.argv[1:]] -if addresses[0] != addresses[1]: raise SystemExit(1) -address=addresses[0] -if not address.is_loopback: raise SystemExit(1) -PY +[[ $bind == 127.0.0.1 && $callback_host == 127.0.0.1 ]] || die 'callback bind/host must both be IPv4 loopback 127.0.0.1' write_auth_config() { local credential=$1 method=$2 url=$3 output=$4 payload=${5:-} @@ -100,8 +102,10 @@ if payload: Path(target).write_text('\n'.join(lines) + '\n') PY chmod 600 "$config" - curl --config "$config" + local status=0 + curl --config "$config" || status=$? rm -f "$config" + return "$status" } make_app_jwt() { @@ -168,6 +172,18 @@ for directory in "$etc_dir" "$secret_dir"; do install -d -m 0700 "$directory" fi done +if [[ -f $bootstrap_pending && ! -L $bootstrap_pending && $(stat -c %u "$bootstrap_pending") == "$expected_uid" && $(stat -c %a "$bootstrap_pending") == 600 ]]; then + if python3 - "$bootstrap_pending" <<'PY' +import json,sys +value=json.load(open(sys.argv[1])) +if any(not value.get(key) for key in ('id','client_id','pem','slug')): raise SystemExit(1) +PY + then mv -fT "$bootstrap_pending" "$bootstrap_recovery" + else die "incomplete protected conversion response remains at $bootstrap_pending; do not retry or delete it without owner recovery/abandonment approval" + fi +elif [[ -e $bootstrap_pending || -L $bootstrap_pending ]]; then + die 'pending conversion response has unsafe ownership, mode, or type' +fi if [[ -e $bootstrap_recovery || -L $bootstrap_recovery ]]; then [[ $mode != check ]] || die 'bootstrap credential recovery is pending; rerun live bootstrap first' recover_conversion @@ -201,7 +217,7 @@ PY done [[ -f $code_file ]] || die 'registration callback timed out' conversion_config=$temporary/conversion.conf - python3 - "$code_file" "$bootstrap_recovery" "$conversion_config" <<'PY' + python3 - "$code_file" "$bootstrap_pending" "$conversion_config" <<'PY' from pathlib import Path import sys code_file, output, config = map(Path, sys.argv[1:]) @@ -212,9 +228,18 @@ config.write_text('\n'.join(['silent','show-error','fail-with-body','request = " 'header = "Accept: application/vnd.github+json"','header = "X-GitHub-Api-Version: 2022-11-28"']) + '\n') PY chmod 600 "$conversion_config" - if ! curl --config "$conversion_config"; then rm -f "$bootstrap_recovery"; die 'manifest conversion failed'; fi - chmod 600 "$bootstrap_recovery" + conversion_status=0 + curl --config "$conversion_config" || conversion_status=$? + chmod 600 "$bootstrap_pending" rm -f "$code_file" "$conversion_config" + if python3 - "$bootstrap_pending" <<'PY' +import json,sys +value=json.load(open(sys.argv[1])) +if any(not value.get(key) for key in ('id','client_id','pem','slug')): raise SystemExit(1) +PY + then mv -fT "$bootstrap_pending" "$bootstrap_recovery" + else die "manifest conversion did not produce a complete response (curl status $conversion_status); protected pending bytes retained for owner recovery" + fi if [[ ${CI_FLEET_TEST_FAIL_AFTER_CONVERSION:-0} == 1 && ${CI_FLEET_TESTING:-0} == 1 ]]; then die 'injected failure after manifest conversion'; fi recover_conversion fi @@ -295,29 +320,17 @@ PY rm -f "$installation_response" repository_ids=$temporary/repository-ids -: >"$repository_ids" -for repository in "${repositories[@]}"; do - response=$temporary/repository.json - write_auth_config "$installation_token" GET "https://api.github.com/repos/$repository" "$response" - python3 - "$response" "$organization" "$repository" >>"$repository_ids" <<'PY' -import json,sys -v=json.load(open(sys.argv[1])) -if v.get('full_name','').lower()!=sys.argv[3].lower() or v.get('owner',{}).get('login','').lower()!=sys.argv[2].lower() or not v.get('private') or v.get('archived'): raise SystemExit(1) -print(v['id']) -PY - rm -f "$response" -done -sort -n -u -o "$repository_ids" "$repository_ids" +printf '%s\n' "${repository_id_values[@]}" | sort -n -u >"$repository_ids" +(($(wc -l <"$repository_ids") == ${#repository_id_values[@]})) || die 'duplicate project repository IDs are forbidden' visible=$temporary/visible.json write_auth_config "$installation_token" GET 'https://api.github.com/installation/repositories?per_page=100' "$visible" -python3 - "$visible" "${repositories[@]}" <<'PY' || die 'App installation repository access is broader or narrower than requested' +python3 - "$visible" "$config_repository" <<'PY' || die 'App installation must select only the private configuration repository' import json,sys value=json.load(open(sys.argv[1])); repositories=value.get('repositories',[]) if value.get('total_count') != len(repositories): raise SystemExit(1) -actual=sorted(v['full_name'].lower() for v in repositories) -expected=sorted(v.lower() for v in sys.argv[2:]) -if actual != expected: raise SystemExit(1) +actual=[v for v in repositories if v.get('full_name','').lower()==sys.argv[2].lower() and v.get('private') and not v.get('archived')] +if len(repositories)!=1 or len(actual)!=1: raise SystemExit(1) PY rm -f "$visible" @@ -339,7 +352,7 @@ if [[ -z $group_id ]]; then import json,sys from pathlib import Path ids=[int(v) for v in Path(sys.argv[3]).read_text().splitlines()] -Path(sys.argv[1]).write_text(json.dumps({'name':sys.argv[2],'visibility':'selected','allows_public_repositories':False,'selected_repository_ids':ids})) +Path(sys.argv[1]).write_text(json.dumps({'name':sys.argv[2],'visibility':'selected','allows_public_repositories':False,'restricted_to_workflows':False,'selected_repository_ids':ids})) PY response=$temporary/group.json write_auth_config "$installation_token" POST "https://api.github.com/orgs/$organization/actions/runner-groups" "$response" "$payload" @@ -356,18 +369,19 @@ write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organi python3 - "$response" "$runner_group" <<'PY' || die 'existing runner group is broader or has unexpected identity; no change made' import json,sys v=json.load(open(sys.argv[1])) -if v.get('name')!=sys.argv[2] or v.get('visibility')!='selected' or v.get('default') or v.get('allows_public_repositories') is not False: raise SystemExit(1) +if v.get('name')!=sys.argv[2] or v.get('visibility')!='selected' or v.get('default') or v.get('allows_public_repositories') is not False or v.get('restricted_to_workflows') is not False: raise SystemExit(1) PY rm -f "$response" selected=$temporary/group-repositories.json write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id/repositories?per_page=100" "$selected" -python3 - "$selected" "$repository_ids" <<'PY' || die 'existing runner group repository access differs; no change made' +python3 - "$selected" "$repository_ids" "${repository_names[@]}" <<'PY' || die 'existing runner group repository access differs; no change made' import json,sys from pathlib import Path response=json.load(open(sys.argv[1])); repositories=response.get('repositories',[]) if response.get('total_count') != len(repositories): raise SystemExit(1) actual=sorted(v['id'] for v in repositories); expected=sorted(int(v) for v in Path(sys.argv[2]).read_text().splitlines()) -if actual!=expected: raise SystemExit(1) +actual_names=sorted(v['full_name'].lower() for v in repositories); expected_names=sorted(v.lower() for v in sys.argv[3:]) +if actual!=expected or actual_names!=expected_names: raise SystemExit(1) PY rm -f "$selected" diff --git a/scripts/fixtures/fake-bootstrap-curl.py b/scripts/fixtures/fake-bootstrap-curl.py index eb4755b..7281bb5 100755 --- a/scripts/fixtures/fake-bootstrap-curl.py +++ b/scripts/fixtures/fake-bootstrap-curl.py @@ -38,29 +38,37 @@ "owner": {"login": "example-org"}} elif "/installation/repositories" in url: kind = "installation-repositories" - response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True}]} + response = {"total_count": 1, "repositories": [{"id": 201, "full_name": "example-org/config", "private": True, "archived": False}]} elif endpoint.endswith("/actions/runner-groups") and method == "POST": kind = "runner-group-create" state.write_text("created\n") - response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False} + response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False, "restricted_to_workflows": False} elif endpoint.endswith("/actions/runner-groups"): kind = "runner-groups" - groups = [{"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False}] if state.exists() else [] + groups = [{"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False, "restricted_to_workflows": False}] if state.exists() else [] response = {"total_count": len(groups), "runner_groups": groups} elif endpoint.endswith("/actions/runner-groups/789/repositories"): kind = "runner-group-repositories" response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True}]} elif endpoint.endswith("/actions/runner-groups/789"): kind = "runner-group" - response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False} + response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False, "restricted_to_workflows": False} else: raise SystemExit(f"unexpected fake API request: {method} {url}") if kind == "installations" and os.environ.get("FAKE_BOOTSTRAP_ALL_REPOSITORIES") == "1": response[0]["repository_selection"] = "all" if kind in {"runner-group", "runner-groups"} and os.environ.get("FAKE_BOOTSTRAP_PUBLIC_GROUP") == "1": (response["runner_groups"][0] if kind == "runner-groups" else response)["allows_public_repositories"] = True +if kind in {"runner-group", "runner-groups"} and os.environ.get("FAKE_BOOTSTRAP_RESTRICTED_GROUP") == "1": + (response["runner_groups"][0] if kind == "runner-groups" else response)["restricted_to_workflows"] = True if kind == os.environ.get("FAKE_BOOTSTRAP_TRUNCATE_KIND"): response["total_count"] += 1 +transient = state.with_suffix(".transient") +if kind == "installations" and os.environ.get("FAKE_BOOTSTRAP_TRANSIENT_INSTALLATIONS") == "1" and not transient.exists(): + transient.write_text("failed-once\n") + raise SystemExit(22) with Path(os.environ["FAKE_BOOTSTRAP_LOG"]).open("a") as log: log.write(f"{method} {kind}\n") output.write_text(json.dumps(response)) +if kind == "manifest-conversion" and os.environ.get("FAKE_BOOTSTRAP_CONVERSION_NONZERO") == "1": + raise SystemExit(22) diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh index eafcb17..e23c63e 100755 --- a/scripts/test-bootstrap-github.sh +++ b/scripts/test-bootstrap-github.sh @@ -45,15 +45,19 @@ grep -Fq 'https://github.com/apps/example-fixture/installations/new' "$tmp/next. # Dry-run validates a complete fictional request and performs no write or API call. dry_root=$tmp/dry-root CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$dry_root "$bootstrap" --dry-run --organization example-org --instance example-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo >"$tmp/dry.out" + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >"$tmp/dry.out" grep -Fq 'NO_GITHUB_MUTATION' "$tmp/dry.out" || fail 'dry-run omitted its no-mutation result' [[ ! -e $dry_root ]] || fail 'dry-run wrote host state' if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group Default --allow-repository example-org/example-repo >/dev/null 2>&1; then fail 'default runner group was accepted' fi -if "$bootstrap" --dry-run --organization example-org --instance example.ci --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then +if "$bootstrap" --dry-run --organization example-org --instance example.ci --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'instance outside the installer contract was accepted' fi +if "$bootstrap" --dry-run --organization Example-org --instance example-ci-01 --config-repository Example-org/config --runner-group example-ci-experimental --allow-repository Example-org/example-repo=101 >/dev/null 2>&1; then fail 'mixed-case organization was accepted'; fi +if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group example_group --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'runner group outside schema-v3 slug contract was accepted'; fi +if "$bootstrap" --dry-run --organization example-organization --instance example-instance --config-repository example-organization/config --runner-group example-group --allow-repository example-organization/repo=101 >/dev/null 2>&1; then fail 'overlong generated App name was accepted'; fi +if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --bind ::1 --callback-host ::1 >/dev/null 2>&1; then fail 'unsupported IPv6 callback was accepted'; fi if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group example-ci-experimental \ --allow-repository example-org/example-repo --bind 10.0.0.1 --callback-host 10.0.0.1 >/dev/null 2>&1; then fail 'plaintext non-loopback callback was accepted' @@ -66,7 +70,7 @@ unsafe_root=$tmp/unsafe-root mkdir -p "$unsafe_root/etc" "$tmp/redirected" ln -s "$tmp/redirected" "$unsafe_root/etc/ci-fleet" if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$unsafe_root "$bootstrap" --check --organization example-org --instance example-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'symlinked protected directory was accepted' fi @@ -80,8 +84,8 @@ openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "$tmp/fixture. export FAKE_BOOTSTRAP_STATE=$tmp/group-created FAKE_BOOTSTRAP_PEM_FILE=$tmp/fixture.pem FAKE_BOOTSTRAP_LOG=$tmp/api.log live_root=$tmp/live-root live_port=$(port) -PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_TEST_FAIL_AFTER_CONVERSION=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo --port "$live_port" --timeout 30 >"$tmp/fault.out" 2>"$tmp/fault.err" & +PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_TEST_FAIL_AFTER_CONVERSION=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_CONVERSION_NONZERO=1 "$bootstrap" --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --port "$live_port" --timeout 30 >"$tmp/fault.out" 2>"$tmp/fault.err" & live_pid=$! processes+=("$live_pid") wait_http "http://127.0.0.1:$live_port/" "$tmp/live.html" @@ -97,8 +101,8 @@ PY "http://127.0.0.1:$live_port/callback" -o "$tmp/live-callback.html" if wait "$live_pid"; then fail 'injected post-conversion failure succeeded'; fi [[ -f $live_root/etc/ci-fleet/bootstrap-recovery.json && ! -e $live_root/etc/ci-fleet/secrets/github-app.pem ]] || fail 'converted credentials were not left recoverable' -PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo --port "$live_port" --timeout 30 >"$tmp/live.out" 2>"$tmp/live.err" +PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_TRANSIENT_INSTALLATIONS=1 "$bootstrap" --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --port "$live_port" --timeout 30 >"$tmp/live.out" 2>"$tmp/live.err" grep -Fq 'BOOTSTRAP_OK organization=example-org instance=example-ci-01 app=ci-fleet-example-org-example-ci-01 app_id=123 installation_id=456 runner_group=example-ci-experimental runner_group_id=789 repositories=1' "$tmp/live.out" || fail 'live bootstrap did not complete' host_env=$live_root/etc/ci-fleet/host.env pem=$live_root/etc/ci-fleet/secrets/github-app.pem @@ -113,20 +117,22 @@ grep -Fxq 'POST runner-group-create' "$tmp/api.log" || fail 'runner group was no before=$(sha256sum "$host_env" "$pem") PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --check --organization example-org --instance example-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo >"$tmp/check.out" 2>"$tmp/check.err" + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >"$tmp/check.out" 2>"$tmp/check.err" [[ $(sha256sum "$host_env" "$pem") == "$before" ]] || fail 'check mode changed credentials' grep -Fq 'CREDENTIALS_VERIFIED' "$tmp/check.out" || fail 'check mode did not verify persisted credentials' [[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == 1 ]] || fail 'check mode mutated the runner group' if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --check --organization example-org --instance other-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then fail 'persisted App was rebound to another instance'; fi + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'persisted App was rebound to another instance'; fi if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_ALL_REPOSITORIES=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then fail 'all-repository installation was accepted'; fi + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'all-repository installation was accepted'; fi if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_PUBLIC_GROUP=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then fail 'public-repository runner group was accepted'; fi + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'public-repository runner group was accepted'; fi +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_RESTRICTED_GROUP=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'workflow-restricted runner group was accepted'; fi for kind in installation-repositories runner-groups runner-group-repositories; do if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_TRUNCATE_KIND=$kind "$bootstrap" --check --organization example-org --instance example-ci-01 \ - --runner-group example-ci-experimental --allow-repository example-org/example-repo >/dev/null 2>&1; then fail "truncated $kind page was accepted"; fi + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail "truncated $kind page was accepted"; fi done printf 'BOOTSTRAP_TESTS_OK\n' From 4fbb18468360b44ca091d7cf59b5e32360818ba7 Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sun, 16 Aug 2026 03:27:27 -0500 Subject: [PATCH 04/10] fix: close GitHub bootstrap verification gaps --- scripts/bootstrap-github.sh | 27 ++++++++++++++++++++++--- scripts/fixtures/fake-bootstrap-curl.py | 10 +++++++-- scripts/test-bootstrap-github.sh | 11 ++++++++++ 3 files changed, 43 insertions(+), 5 deletions(-) diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh index 34ec11d..2c4b679 100755 --- a/scripts/bootstrap-github.sh +++ b/scripts/bootstrap-github.sh @@ -58,6 +58,14 @@ if $run_installer; then [[ -n $config_repo && $config_ref =~ ^[0-9a-f]{40}$ ]] || die '--install requires --config-repo and a 40-character --config-ref' fi for command in bash curl openssl python3 install mktemp stat; do command -v "$command" >/dev/null || die "required command is unavailable: $command"; done +if $run_installer; then + command -v git >/dev/null || die 'required command is unavailable: git' + if git -C "$config_repo" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + [[ $(git -C "$config_repo" rev-parse "$config_ref^{commit}" 2>/dev/null || true) == "$config_ref" ]] || die 'local configuration repository does not contain --config-ref' + else + [[ ${config_repo,,} == "${config_repository,,}" ]] || die '--config-repo must match --config-repository unless it is a pinned local checkout' + fi +fi script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) root_prefix=${CI_FLEET_ROOT_PREFIX:-} @@ -248,10 +256,10 @@ app_jwt=$temporary/app.jwt make_app_jwt "$CI_FLEET_GITHUB_APP_ID" "$pem" "$app_jwt" app_response=$temporary/app.json write_auth_config "$app_jwt" GET https://api.github.com/app "$app_response" -mapfile -t app_data < <(python3 - "$app_response" "$organization" "$CI_FLEET_GITHUB_APP_SLUG" <<'PY' +mapfile -t app_data < <(python3 - "$app_response" "$organization" "$CI_FLEET_GITHUB_APP_SLUG" "$CI_FLEET_GITHUB_APP_CLIENT_ID" <<'PY' import json,sys v=json.load(open(sys.argv[1])); owner=v.get('owner',{}); expected=sys.argv[3] -if (expected and v.get('slug') != expected) or owner.get('login','').lower() != sys.argv[2].lower() or owner.get('type') != 'Organization': raise SystemExit(1) +if (expected and v.get('slug') != expected) or v.get('client_id') != sys.argv[4] or owner.get('login','').lower() != sys.argv[2].lower() or owner.get('type') != 'Organization': raise SystemExit(1) permissions=v.get('permissions',{}) expected_permissions={'contents':'read','metadata':'read','organization_self_hosted_runners':'write'} if permissions != expected_permissions or v.get('events') not in ([], None) or v.get('public') is not False: raise SystemExit(1) @@ -314,7 +322,7 @@ python3 - "$installation_response" "$installation_token" <<'PY' import json,os,sys from pathlib import Path value=json.load(open(sys.argv[1])); token=value.get('token') -if not isinstance(token,str) or len(token)<20: raise SystemExit(1) +if not isinstance(token,str) or len(token)<20 or value.get('permissions',{}).get('contents') != 'read': raise SystemExit(1) Path(sys.argv[2]).write_text(token); os.chmod(sys.argv[2],0o600) PY rm -f "$installation_response" @@ -334,6 +342,19 @@ if len(repositories)!=1 or len(actual)!=1: raise SystemExit(1) PY rm -f "$visible" +for repository in "${repositories[@]}"; do + repository_name=${repository%=*} + repository_id=${repository##*=} + response=$temporary/repository.json + write_auth_config "$installation_token" GET "https://api.github.com/repos/$repository_name" "$response" + python3 - "$response" "$repository_name" "$repository_id" "$organization" <<'PY' || die "project repository name/ID pair is invalid: $repository_name" +import json,sys +value=json.load(open(sys.argv[1])); owner=value.get('owner',{}) +if str(value.get('id')) != sys.argv[3] or value.get('full_name','').lower() != sys.argv[2].lower() or owner.get('login','').lower() != sys.argv[4].lower() or not value.get('private') or value.get('archived'): raise SystemExit(1) +PY + rm -f "$response" +done + groups=$temporary/groups.json write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups?per_page=100" "$groups" group_id=$(python3 - "$groups" "$runner_group" <<'PY' diff --git a/scripts/fixtures/fake-bootstrap-curl.py b/scripts/fixtures/fake-bootstrap-curl.py index 7281bb5..e5ec2d7 100755 --- a/scripts/fixtures/fake-bootstrap-curl.py +++ b/scripts/fixtures/fake-bootstrap-curl.py @@ -24,14 +24,14 @@ "client_secret": "fixture-client-secret", "webhook_secret": "fixture-webhook-secret"} elif endpoint.endswith("/app"): kind = "app" - response = {"id": 123, "slug": "ci-fleet-example-org-example-ci-01", "public": False, "events": [], "owner": {"login": "example-org", "type": "Organization"}, + response = {"id": 123, "client_id": "Iv1FixtureClient", "slug": "ci-fleet-example-org-example-ci-01", "public": False, "events": [], "owner": {"login": "example-org", "type": "Organization"}, "permissions": {"contents": "read", "metadata": "read", "organization_self_hosted_runners": "write"}} elif endpoint.endswith("/app/installations"): kind = "installations" response = [{"id": 456, "repository_selection": "selected", "account": {"login": "example-org", "type": "Organization"}}] elif endpoint.endswith("/access_tokens"): kind = "installation-token" - response = {"token": "fixture-installation-token-value"} + response = {"token": "fixture-installation-token-value", "permissions": {"contents": "read", "metadata": "read", "organization_self_hosted_runners": "write"}} elif "/repos/example-org/example-repo" in url: kind = "repository" response = {"id": 101, "full_name": "example-org/example-repo", "private": True, "archived": False, @@ -57,6 +57,12 @@ raise SystemExit(f"unexpected fake API request: {method} {url}") if kind == "installations" and os.environ.get("FAKE_BOOTSTRAP_ALL_REPOSITORIES") == "1": response[0]["repository_selection"] = "all" +if kind == "app" and os.environ.get("FAKE_BOOTSTRAP_BAD_APP_CLIENT_ID") == "1": + assert isinstance(response, dict) + response["client_id"] = "Iv1DifferentClient" +if kind == "installation-token" and os.environ.get("FAKE_BOOTSTRAP_NO_CONTENTS") == "1": + assert isinstance(response, dict) + response["permissions"].pop("contents") if kind in {"runner-group", "runner-groups"} and os.environ.get("FAKE_BOOTSTRAP_PUBLIC_GROUP") == "1": (response["runner_groups"][0] if kind == "runner-groups" else response)["allows_public_repositories"] = True if kind in {"runner-group", "runner-groups"} and os.environ.get("FAKE_BOOTSTRAP_RESTRICTED_GROUP") == "1": diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh index e23c63e..497f7b9 100755 --- a/scripts/test-bootstrap-github.sh +++ b/scripts/test-bootstrap-github.sh @@ -58,6 +58,9 @@ if "$bootstrap" --dry-run --organization Example-org --instance example-ci-01 -- if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group example_group --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'runner group outside schema-v3 slug contract was accepted'; fi if "$bootstrap" --dry-run --organization example-organization --instance example-instance --config-repository example-organization/config --runner-group example-group --allow-repository example-organization/repo=101 >/dev/null 2>&1; then fail 'overlong generated App name was accepted'; fi if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --bind ::1 --callback-host ::1 >/dev/null 2>&1; then fail 'unsupported IPv6 callback was accepted'; fi +install_mismatch_root=$tmp/install-mismatch-root +if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$install_mismatch_root "$bootstrap" --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --install --config-repo example-org/other --config-ref 1111111111111111111111111111111111111111 >/dev/null 2>&1; then fail 'mismatched installer configuration repository was accepted'; fi +[[ ! -e $install_mismatch_root ]] || fail 'mismatched installer repository wrote local state before rejection' if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group example-ci-experimental \ --allow-repository example-org/example-repo --bind 10.0.0.1 --callback-host 10.0.0.1 >/dev/null 2>&1; then fail 'plaintext non-loopback callback was accepted' @@ -114,6 +117,10 @@ if grep -R -F -e fixture-conversion-code -e fixture-client-secret -e fixture-web fail 'bootstrap output exposed sensitive fixture material' fi grep -Fxq 'POST runner-group-create' "$tmp/api.log" || fail 'runner group was not created in the mocked live flow' +create_count=$(grep -c '^POST runner-group-create$' "$tmp/api.log") +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=999 >/dev/null 2>&1; then fail 'mismatched project repository name/ID pair was accepted'; fi +[[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == "$create_count" ]] || fail 'invalid repository pair mutated the runner group' before=$(sha256sum "$host_env" "$pem") PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --check --organization example-org --instance example-ci-01 \ @@ -130,6 +137,10 @@ if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAK --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'public-repository runner group was accepted'; fi if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_RESTRICTED_GROUP=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'workflow-restricted runner group was accepted'; fi +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_BAD_APP_CLIENT_ID=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'mismatched App client ID was accepted'; fi +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_NO_CONTENTS=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'installation token without contents permission was accepted'; fi for kind in installation-repositories runner-groups runner-group-repositories; do if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_TRUNCATE_KIND=$kind "$bootstrap" --check --organization example-org --instance example-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail "truncated $kind page was accepted"; fi From e3ea10a39d95b80c51e0d578476f64b9d4cd4ab1 Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sun, 16 Aug 2026 03:55:07 -0500 Subject: [PATCH 05/10] fix: roll back unverified runner groups --- scripts/bootstrap-github.sh | 47 ++++++++++++++++--------- scripts/fixtures/fake-bootstrap-curl.py | 4 +++ scripts/test-bootstrap-github.sh | 7 ++-- 3 files changed, 39 insertions(+), 19 deletions(-) diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh index 2c4b679..832b5fd 100755 --- a/scripts/bootstrap-github.sh +++ b/scripts/bootstrap-github.sh @@ -342,19 +342,6 @@ if len(repositories)!=1 or len(actual)!=1: raise SystemExit(1) PY rm -f "$visible" -for repository in "${repositories[@]}"; do - repository_name=${repository%=*} - repository_id=${repository##*=} - response=$temporary/repository.json - write_auth_config "$installation_token" GET "https://api.github.com/repos/$repository_name" "$response" - python3 - "$response" "$repository_name" "$repository_id" "$organization" <<'PY' || die "project repository name/ID pair is invalid: $repository_name" -import json,sys -value=json.load(open(sys.argv[1])); owner=value.get('owner',{}) -if str(value.get('id')) != sys.argv[3] or value.get('full_name','').lower() != sys.argv[2].lower() or owner.get('login','').lower() != sys.argv[4].lower() or not value.get('private') or value.get('archived'): raise SystemExit(1) -PY - rm -f "$response" -done - groups=$temporary/groups.json write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups?per_page=100" "$groups" group_id=$(python3 - "$groups" "$runner_group" <<'PY' @@ -366,6 +353,7 @@ for value in groups: PY ) rm -f "$groups" +created_group=false if [[ -z $group_id ]]; then [[ $mode != check ]] || die 'runner group is missing' payload=$temporary/group-create.json @@ -382,20 +370,38 @@ import json,sys print(json.load(open(sys.argv[1]))['id']) PY ) + created_group=true rm -f "$payload" "$response" fi [[ $group_id =~ ^[0-9]+$ ]] || die 'runner group ID is invalid' +rollback_new_group() { + $created_group || return 0 + local deleted=$temporary/group-delete.json + write_auth_config "$installation_token" DELETE "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id" "$deleted" || return 1 + rm -f "$deleted" + created_group=false +} response=$temporary/group.json -write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id" "$response" -python3 - "$response" "$runner_group" <<'PY' || die 'existing runner group is broader or has unexpected identity; no change made' +if ! write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id" "$response"; then + if $created_group; then rollback_new_group || die 'new runner group inspection and rollback both failed'; die 'runner group inspection failed; new group was rolled back'; fi + die 'existing runner group inspection failed; no change made' +fi +if ! python3 - "$response" "$runner_group" <<'PY' import json,sys v=json.load(open(sys.argv[1])) if v.get('name')!=sys.argv[2] or v.get('visibility')!='selected' or v.get('default') or v.get('allows_public_repositories') is not False or v.get('restricted_to_workflows') is not False: raise SystemExit(1) PY +then + if $created_group; then rollback_new_group || die 'new runner group failed identity verification and rollback failed'; die 'new runner group failed identity verification and was rolled back'; fi + die 'existing runner group is broader or has unexpected identity; no change made' +fi rm -f "$response" selected=$temporary/group-repositories.json -write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id/repositories?per_page=100" "$selected" -python3 - "$selected" "$repository_ids" "${repository_names[@]}" <<'PY' || die 'existing runner group repository access differs; no change made' +if ! write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id/repositories?per_page=100" "$selected"; then + if $created_group; then rollback_new_group || die 'new runner group repository inspection and rollback both failed'; die 'runner group repository inspection failed; new group was rolled back'; fi + die 'existing runner group repository inspection failed; no change made' +fi +if ! python3 - "$selected" "$repository_ids" "${repository_names[@]}" <<'PY' import json,sys from pathlib import Path response=json.load(open(sys.argv[1])); repositories=response.get('repositories',[]) @@ -404,6 +410,13 @@ actual=sorted(v['id'] for v in repositories); expected=sorted(int(v) for v in Pa actual_names=sorted(v['full_name'].lower() for v in repositories); expected_names=sorted(v.lower() for v in sys.argv[3:]) if actual!=expected or actual_names!=expected_names: raise SystemExit(1) PY +then + if $created_group; then + rollback_new_group || die 'new runner group failed verification and rollback failed' + die 'new runner group repository access differed and was rolled back' + fi + die 'existing runner group repository access differs; no change made' +fi rm -f "$selected" staged=$temporary/host.env diff --git a/scripts/fixtures/fake-bootstrap-curl.py b/scripts/fixtures/fake-bootstrap-curl.py index e5ec2d7..42ec2e3 100755 --- a/scripts/fixtures/fake-bootstrap-curl.py +++ b/scripts/fixtures/fake-bootstrap-curl.py @@ -50,6 +50,10 @@ elif endpoint.endswith("/actions/runner-groups/789/repositories"): kind = "runner-group-repositories" response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True}]} +elif endpoint.endswith("/actions/runner-groups/789") and method == "DELETE": + kind = "runner-group-delete" + state.unlink(missing_ok=True) + response = {} elif endpoint.endswith("/actions/runner-groups/789"): kind = "runner-group" response = {"id": 789, "name": "example-ci-experimental", "visibility": "selected", "default": False, "allows_public_repositories": False, "restricted_to_workflows": False} diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh index 497f7b9..9be23de 100755 --- a/scripts/test-bootstrap-github.sh +++ b/scripts/test-bootstrap-github.sh @@ -118,16 +118,19 @@ if grep -R -F -e fixture-conversion-code -e fixture-client-secret -e fixture-web fi grep -Fxq 'POST runner-group-create' "$tmp/api.log" || fail 'runner group was not created in the mocked live flow' create_count=$(grep -c '^POST runner-group-create$' "$tmp/api.log") +rm -f "$FAKE_BOOTSTRAP_STATE" if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=999 >/dev/null 2>&1; then fail 'mismatched project repository name/ID pair was accepted'; fi -[[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == "$create_count" ]] || fail 'invalid repository pair mutated the runner group' +[[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == "$((create_count + 1))" && $(grep -c '^DELETE runner-group-delete$' "$tmp/api.log") == 1 && ! -e $FAKE_BOOTSTRAP_STATE ]] || fail 'invalid repository pair was not rolled back' +printf 'created\n' >"$FAKE_BOOTSTRAP_STATE" +post_rollback_create_count=$(grep -c '^POST runner-group-create$' "$tmp/api.log") before=$(sha256sum "$host_env" "$pem") PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --check --organization example-org --instance example-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >"$tmp/check.out" 2>"$tmp/check.err" [[ $(sha256sum "$host_env" "$pem") == "$before" ]] || fail 'check mode changed credentials' grep -Fq 'CREDENTIALS_VERIFIED' "$tmp/check.out" || fail 'check mode did not verify persisted credentials' -[[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == 1 ]] || fail 'check mode mutated the runner group' +[[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == "$post_rollback_create_count" ]] || fail 'check mode mutated the runner group' if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --check --organization example-org --instance other-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'persisted App was rebound to another instance'; fi From 7255a7c3239db811782a560bb8c1874340f25306 Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sun, 16 Aug 2026 04:14:56 -0500 Subject: [PATCH 06/10] fix: verify bootstrap routing before mutation --- docs/GITHUB-APP-SETUP.md | 31 +++++++++------ scripts/bootstrap-github.sh | 52 +++++++++++++++++-------- scripts/fixtures/fake-bootstrap-curl.py | 5 ++- scripts/test-bootstrap-github.sh | 26 +++++++++++++ 4 files changed, 85 insertions(+), 29 deletions(-) diff --git a/docs/GITHUB-APP-SETUP.md b/docs/GITHUB-APP-SETUP.md index 06325fb..f9903f2 100644 --- a/docs/GITHUB-APP-SETUP.md +++ b/docs/GITHUB-APP-SETUP.md @@ -60,7 +60,9 @@ The bootstrap: - rejects public, archived, wrong-organization, broader App installation, and default/broad runner-group access; - creates a missing selected-repository runner group, but never changes an - existing group whose identity or access differs; + existing group whose identity or access differs. A group created by the + current invocation is automatically deleted if inspection, routing + verification, or cancellation occurs before verification completes; - destroys the conversion code, JWTs, installation token, callback state, and temporary curl configurations on exit. After a successful conversion, a protected mode-`0600` `bootstrap-recovery.json` (or atomic-publication @@ -84,11 +86,15 @@ After success, rerun the same command with `--check` to verify App ownership, permissions, exact selected private repositories, exact runner-group access, and host-local credential modes without changing the App or runner group. -Pass `--install --config-repo OWNER/REPO --config-ref REVIEWED_COMMIT` on the -initial live command for direct handoff to the idempotent host installer. Without -it, the redacted final report prints the exact installer shape. Externally -provisioned credentials remain supported; the generic installer never requires -this bootstrap on every run. +Pass `--install --config-repo /PATH/TO/PINNED/CHECKOUT --config-ref REVIEWED_COMMIT` +on the initial live command for direct handoff to the idempotent host installer. +Bootstrap requires the commit in that local checkout and verifies that its +selected controller pool exactly matches the requested organization, runner +group, and project allowlist before any GitHub mutation. Remote `OWNER/REPO` +handoff is rejected because a fresh host has no independently verified Git +credential before App creation. Without `--install`, the redacted final report +prints the installer shape. Externally provisioned credentials remain supported; +the generic installer never requires this bootstrap on every run. The controller exchanges a short-lived JWT signed with the PEM for an installation token at runtime (`scripts/github-app-token.sh`). No token is @@ -135,14 +141,17 @@ sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh --check-only ## Accidental creation rollback -The bootstrap never replaces or broadens an existing App or runner group. If a -new object was approved accidentally, stop before running the installer. An +The bootstrap never replaces or broadens an existing App or runner group. A new +runner group is automatically deleted only when the same invocation cannot +verify its identity/access or is cancelled before verification; it never deletes +a pre-existing group. If another new object was approved accidentally, stop +before running the installer. An organization owner must compare the App slug/ID and group name in the redacted bootstrap report with GitHub's settings, verify the new group has no runners, then remove only those exact newly created objects. Preserve and investigate any -pre-existing or mismatched object. App/group deletion and owner approval are live -GitHub-setting mutations and therefore require separate authorization; this -repository command does not automate them. +pre-existing or mismatched object. App deletion and deletion of any group outside +the bounded automatic rollback above are live GitHub-setting mutations and +therefore require separate authorization; this command does not automate them. ## Rotation diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh index 832b5fd..9184bd3 100755 --- a/scripts/bootstrap-github.sh +++ b/scripts/bootstrap-github.sh @@ -8,7 +8,7 @@ Usage: bootstrap-github.sh [--dry-run|--check] --organization ORG --instance ID --config-repository ORG/PRIVATE_CONFIG_REPO --runner-group GROUP --allow-repository ORG/PROJECT=NUMERIC_ID [--allow-repository ORG/PROJECT=NUMERIC_ID ...] [--bind ADDRESS --callback-host HOST --port PORT --timeout SECONDS] - [--install --config-repo OWNER/REPO_OR_PATH --config-ref COMMIT] + [--install --config-repo LOCAL_CHECKOUT --config-ref COMMIT] EOF } die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } @@ -60,11 +60,17 @@ fi for command in bash curl openssl python3 install mktemp stat; do command -v "$command" >/dev/null || die "required command is unavailable: $command"; done if $run_installer; then command -v git >/dev/null || die 'required command is unavailable: git' - if git -C "$config_repo" rev-parse --is-inside-work-tree >/dev/null 2>&1; then - [[ $(git -C "$config_repo" rev-parse "$config_ref^{commit}" 2>/dev/null || true) == "$config_ref" ]] || die 'local configuration repository does not contain --config-ref' - else - [[ ${config_repo,,} == "${config_repository,,}" ]] || die '--config-repo must match --config-repository unless it is a pinned local checkout' - fi + git -C "$config_repo" rev-parse --is-inside-work-tree >/dev/null 2>&1 || die '--install requires a local configuration checkout; remote handoff has no pre-mutation credential proof' + [[ $(git -C "$config_repo" rev-parse "$config_ref^{commit}" 2>/dev/null || true) == "$config_ref" ]] || die 'local configuration repository does not contain --config-ref' + git -C "$config_repo" show "$config_ref:fleet.json" | python3 -c ' +import json,sys +organization, controller_id, runner_group = sys.argv[1:4] +repositories=sorted(item.lower() for item in sys.argv[4:]) +value=json.load(sys.stdin) +controller=value["controllers"][controller_id] +pool=value["runner_pools"][controller["pool"]] +if value["organization"]["slug"] != organization or pool["runner_group"] != runner_group or pool.get("public_repositories") is not False or sorted(item.lower() for item in pool["allowed_repositories"]) != repositories: raise SystemExit(1) +' "$organization" "$instance" "$runner_group" "${repository_names[@]}" || die 'pinned desired-state routing differs from bootstrap arguments' fi script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) @@ -80,16 +86,25 @@ bootstrap_recovery=$etc_dir/bootstrap-recovery.json bootstrap_pending=$etc_dir/bootstrap-recovery.pending temporary=$(mktemp -d) callback_pid= +created_group=false +installation_token= +group_id= expected_uid=0 [[ ${CI_FLEET_TESTING:-0} != 1 ]] || expected_uid=$(id -u) cleanup() { local status=$? - trap - EXIT INT TERM HUP + trap - EXIT HUP INT TERM [[ -z $callback_pid ]] || kill "$callback_pid" >/dev/null 2>&1 || true + if $created_group && declare -F rollback_new_group >/dev/null; then + rollback_new_group || printf 'ERROR: automatic rollback of the unverified runner group failed\n' >&2 + fi rm -rf "$temporary" exit "$status" } -trap cleanup EXIT INT TERM HUP +trap cleanup EXIT +trap 'exit 129' HUP +trap 'exit 130' INT +trap 'exit 143' TERM if [[ $mode != dry-run && ${CI_FLEET_TESTING:-0} != 1 && ${EUID:-$(id -u)} -ne 0 ]]; then die 'run live/check bootstrap as root'; fi [[ $bind == 127.0.0.1 && $callback_host == 127.0.0.1 ]] || die 'callback bind/host must both be IPv4 loopback 127.0.0.1' @@ -116,6 +131,14 @@ PY return "$status" } +rollback_new_group() { + $created_group || return 0 + local deleted=$temporary/group-delete.json + write_auth_config "$installation_token" DELETE "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id" "$deleted" || return 1 + rm -f "$deleted" + created_group=false +} + make_app_jwt() { local app_id=$1 key=$2 output=$3 now header payload now=$(date +%s) @@ -371,16 +394,10 @@ print(json.load(open(sys.argv[1]))['id']) PY ) created_group=true + if [[ ${CI_FLEET_TEST_CANCEL_AFTER_GROUP_CREATE:-0} == 1 && ${CI_FLEET_TESTING:-0} == 1 ]]; then kill -TERM "$$"; fi rm -f "$payload" "$response" fi [[ $group_id =~ ^[0-9]+$ ]] || die 'runner group ID is invalid' -rollback_new_group() { - $created_group || return 0 - local deleted=$temporary/group-delete.json - write_auth_config "$installation_token" DELETE "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id" "$deleted" || return 1 - rm -f "$deleted" - created_group=false -} response=$temporary/group.json if ! write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id" "$response"; then if $created_group; then rollback_new_group || die 'new runner group inspection and rollback both failed'; die 'runner group inspection failed; new group was rolled back'; fi @@ -408,7 +425,7 @@ response=json.load(open(sys.argv[1])); repositories=response.get('repositories', if response.get('total_count') != len(repositories): raise SystemExit(1) actual=sorted(v['id'] for v in repositories); expected=sorted(int(v) for v in Path(sys.argv[2]).read_text().splitlines()) actual_names=sorted(v['full_name'].lower() for v in repositories); expected_names=sorted(v.lower() for v in sys.argv[3:]) -if actual!=expected or actual_names!=expected_names: raise SystemExit(1) +if actual!=expected or actual_names!=expected_names or any(v.get('private') is not True or v.get('archived') is not False for v in repositories): raise SystemExit(1) PY then if $created_group; then @@ -418,6 +435,7 @@ then die 'existing runner group repository access differs; no change made' fi rm -f "$selected" +created_group=false staged=$temporary/host.env printf '%s\n' "CI_FLEET_GITHUB_APP_CLIENT_ID=$CI_FLEET_GITHUB_APP_CLIENT_ID" "CI_FLEET_GITHUB_APP_INSTALLATION_ID=$installation_id" "CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE=/etc/ci-fleet/secrets/github-app.pem" 'CI_FLEET_RUNNER_TTL=6h' >"$staged" @@ -429,7 +447,7 @@ fi note "BOOTSTRAP_OK organization=$organization instance=$instance app=$CI_FLEET_GITHUB_APP_SLUG app_id=$verified_app_id installation_id=$installation_id runner_group=$runner_group runner_group_id=$group_id repositories=${#repositories[@]}" if [[ $mode == check ]]; then note "CREDENTIALS_VERIFIED pem=$pem host_config=$host_env"; else note "CREDENTIALS_WRITTEN pem=$pem host_config=$host_env"; fi if $run_installer; then - "$script_dir/install-worker-controller.sh" --install --config-repo "$config_repo" --controller "$instance" --ref "$config_ref" + "$script_dir/install-worker-controller.sh" --install --config-repo "$config_repo" --config-identity "$config_repository" --controller "$instance" --ref "$config_ref" else note "NEXT sudo $script_dir/install-worker-controller.sh --install --config-repo OWNER/REPO --controller $instance --ref REVIEWED_COMMIT" fi diff --git a/scripts/fixtures/fake-bootstrap-curl.py b/scripts/fixtures/fake-bootstrap-curl.py index 42ec2e3..f142c4d 100755 --- a/scripts/fixtures/fake-bootstrap-curl.py +++ b/scripts/fixtures/fake-bootstrap-curl.py @@ -49,7 +49,7 @@ response = {"total_count": len(groups), "runner_groups": groups} elif endpoint.endswith("/actions/runner-groups/789/repositories"): kind = "runner-group-repositories" - response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True}]} + response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True, "archived": False}]} elif endpoint.endswith("/actions/runner-groups/789") and method == "DELETE": kind = "runner-group-delete" state.unlink(missing_ok=True) @@ -71,6 +71,9 @@ (response["runner_groups"][0] if kind == "runner-groups" else response)["allows_public_repositories"] = True if kind in {"runner-group", "runner-groups"} and os.environ.get("FAKE_BOOTSTRAP_RESTRICTED_GROUP") == "1": (response["runner_groups"][0] if kind == "runner-groups" else response)["restricted_to_workflows"] = True +if kind == "runner-group-repositories" and os.environ.get("FAKE_BOOTSTRAP_ARCHIVED_REPOSITORY") == "1": + assert isinstance(response, dict) + response["repositories"][0]["archived"] = True if kind == os.environ.get("FAKE_BOOTSTRAP_TRUNCATE_KIND"): response["total_count"] += 1 transient = state.with_suffix(".transient") diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh index 9be23de..1a4d855 100755 --- a/scripts/test-bootstrap-github.sh +++ b/scripts/test-bootstrap-github.sh @@ -61,6 +61,26 @@ if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 -- install_mismatch_root=$tmp/install-mismatch-root if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$install_mismatch_root "$bootstrap" --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --install --config-repo example-org/other --config-ref 1111111111111111111111111111111111111111 >/dev/null 2>&1; then fail 'mismatched installer configuration repository was accepted'; fi [[ ! -e $install_mismatch_root ]] || fail 'mismatched installer repository wrote local state before rejection' +config_checkout=$tmp/config-checkout +git init -q "$config_checkout" +python3 - "$config_checkout/fleet.json" <<'PY' +import json,sys +json.dump({'organization':{'slug':'example-org'},'controllers':{'example-ci-01':{'pool':'trusted'}},'runner_pools':{'trusted':{'runner_group':'example-ci-experimental','allowed_repositories':['example-org/example-repo'],'public_repositories':False}}},open(sys.argv[1],'w')) +PY +git -C "$config_checkout" add fleet.json +git -C "$config_checkout" -c user.name=fixture -c user.email=fixture@example.invalid commit -qm fixture +config_ref=$(git -C "$config_checkout" rev-parse HEAD) +routing_mismatch_root=$tmp/routing-mismatch-root +if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$routing_mismatch_root "$bootstrap" --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group other-group --allow-repository example-org/example-repo=101 --install --config-repo "$config_checkout" --config-ref "$config_ref" >/dev/null 2>&1; then fail 'desired-state routing mismatch was accepted'; fi +[[ ! -e $routing_mismatch_root ]] || fail 'desired-state routing mismatch wrote local state before rejection' +routing_root=$tmp/routing-root +routing_port=$(port) +CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$routing_root "$bootstrap" --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --install --config-repo "$config_checkout" --config-ref "$config_ref" --port "$routing_port" --timeout 30 >"$tmp/routing.out" 2>"$tmp/routing.err" & +routing_pid=$! +processes+=("$routing_pid") +wait_http "http://127.0.0.1:$routing_port/" "$tmp/routing.html" +kill -TERM "$routing_pid" +if wait "$routing_pid"; then fail 'cancelled matching desired-state bootstrap succeeded'; fi if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group example-ci-experimental \ --allow-repository example-org/example-repo --bind 10.0.0.1 --callback-host 10.0.0.1 >/dev/null 2>&1; then fail 'plaintext non-loopback callback was accepted' @@ -122,6 +142,10 @@ rm -f "$FAKE_BOOTSTRAP_STATE" if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=999 >/dev/null 2>&1; then fail 'mismatched project repository name/ID pair was accepted'; fi [[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == "$((create_count + 1))" && $(grep -c '^DELETE runner-group-delete$' "$tmp/api.log") == 1 && ! -e $FAKE_BOOTSTRAP_STATE ]] || fail 'invalid repository pair was not rolled back' +cancel_delete_count=$(grep -c '^DELETE runner-group-delete$' "$tmp/api.log") +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_TEST_CANCEL_AFTER_GROUP_CREATE=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'injected cancellation after group creation succeeded'; fi +[[ $(grep -c '^DELETE runner-group-delete$' "$tmp/api.log") == "$((cancel_delete_count + 1))" && ! -e $FAKE_BOOTSTRAP_STATE ]] || fail 'cancellation did not roll back the new runner group' printf 'created\n' >"$FAKE_BOOTSTRAP_STATE" post_rollback_create_count=$(grep -c '^POST runner-group-create$' "$tmp/api.log") @@ -140,6 +164,8 @@ if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAK --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'public-repository runner group was accepted'; fi if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_RESTRICTED_GROUP=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'workflow-restricted runner group was accepted'; fi +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_ARCHIVED_REPOSITORY=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'archived project repository was accepted'; fi if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_BAD_APP_CLIENT_ID=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'mismatched App client ID was accepted'; fi if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_NO_CONTENTS=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ From 73ab4655346236d362512d97bd418da766676e1c Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sun, 16 Aug 2026 04:35:00 -0500 Subject: [PATCH 07/10] fix: complete bootstrap install preflight --- scripts/bootstrap-github.sh | 81 ++++++++++++++++++++++---------- scripts/test-bootstrap-github.sh | 14 +++--- 2 files changed, 62 insertions(+), 33 deletions(-) diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh index 9184bd3..3c9da75 100755 --- a/scripts/bootstrap-github.sh +++ b/scripts/bootstrap-github.sh @@ -58,22 +58,68 @@ if $run_installer; then [[ -n $config_repo && $config_ref =~ ^[0-9a-f]{40}$ ]] || die '--install requires --config-repo and a 40-character --config-ref' fi for command in bash curl openssl python3 install mktemp stat; do command -v "$command" >/dev/null || die "required command is unavailable: $command"; done +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +temporary=$(mktemp -d) +callback_pid= +created_group=false +installation_token= +group_id= +callback_is_running() { + local pid + while read -r pid; do [[ $pid == "$callback_pid" ]] && return 0; done < <(jobs -pr) + return 1 +} +reap_callback_if_exited() { + [[ -n $callback_pid ]] || return 0 + callback_is_running && return 0 + wait "$callback_pid" >/dev/null 2>&1 || true + callback_pid= +} +cleanup() { + local status=$? + trap - EXIT HUP INT TERM + if [[ -n $callback_pid ]]; then + if callback_is_running; then kill "$callback_pid" >/dev/null 2>&1 || true; fi + wait "$callback_pid" >/dev/null 2>&1 || true + callback_pid= + fi + if $created_group && declare -F rollback_new_group >/dev/null; then + rollback_new_group || printf 'ERROR: automatic rollback of the unverified runner group failed\n' >&2 + fi + rm -rf "$temporary" + exit "$status" +} +trap cleanup EXIT +trap 'exit 129' HUP +trap 'exit 130' INT +trap 'exit 143' TERM if $run_installer; then command -v git >/dev/null || die 'required command is unavailable: git' git -C "$config_repo" rev-parse --is-inside-work-tree >/dev/null 2>&1 || die '--install requires a local configuration checkout; remote handoff has no pre-mutation credential proof' [[ $(git -C "$config_repo" rev-parse "$config_ref^{commit}" 2>/dev/null || true) == "$config_ref" ]] || die 'local configuration repository does not contain --config-ref' - git -C "$config_repo" show "$config_ref:fleet.json" | python3 -c ' + candidate_config=$temporary/fleet.json + tree_paths=$temporary/config-tree-paths + git -C "$config_repo" show "$config_ref:fleet.json" >"$candidate_config" || die 'fleet.json is absent at the requested configuration commit' + git -C "$config_repo" ls-tree -rz --name-only "$config_ref" >"$tree_paths" || die 'cannot inspect the configuration commit tree' + validator_args=(--config "$candidate_config" --strict --tree-paths "$tree_paths") + if git -C "$config_repo" cat-file -e "$config_ref:engine-rollout-evidence.json" 2>/dev/null; then + rollout_evidence=$temporary/engine-rollout-evidence.json + git -C "$config_repo" show "$config_ref:engine-rollout-evidence.json" >"$rollout_evidence" || die 'cannot read engine rollout evidence' + validator_args+=(--rollout-evidence "$rollout_evidence") + fi + python3 "$script_dir/../templates/config-repository/scripts/validate.py" "${validator_args[@]}" || die 'pinned configuration commit validation failed' + python3 "$script_dir/scan_committed_secrets.py" --repository "$config_repo" --commit "$config_ref" || die 'pinned configuration commit secret scan failed' + python3 - "$candidate_config" "$organization" "$instance" "$runner_group" "${repository_names[@]}" <<'PY' || die 'pinned desired-state routing differs from bootstrap arguments' import json,sys -organization, controller_id, runner_group = sys.argv[1:4] -repositories=sorted(item.lower() for item in sys.argv[4:]) -value=json.load(sys.stdin) +organization, controller_id, runner_group = sys.argv[2:5] +repositories=sorted(item.lower() for item in sys.argv[5:]) +value=json.load(open(sys.argv[1])) controller=value["controllers"][controller_id] pool=value["runner_pools"][controller["pool"]] if value["organization"]["slug"] != organization or pool["runner_group"] != runner_group or pool.get("public_repositories") is not False or sorted(item.lower() for item in pool["allowed_repositories"]) != repositories: raise SystemExit(1) -' "$organization" "$instance" "$runner_group" "${repository_names[@]}" || die 'pinned desired-state routing differs from bootstrap arguments' +PY fi -script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) root_prefix=${CI_FLEET_ROOT_PREFIX:-} if [[ -n $root_prefix && ${CI_FLEET_TESTING:-0} != 1 ]]; then die 'CI_FLEET_ROOT_PREFIX is test-only'; fi root_path() { printf '%s%s' "$root_prefix" "$1"; } @@ -84,27 +130,8 @@ host_env=$etc_dir/host.env bootstrap_state=$etc_dir/bootstrap-app.env bootstrap_recovery=$etc_dir/bootstrap-recovery.json bootstrap_pending=$etc_dir/bootstrap-recovery.pending -temporary=$(mktemp -d) -callback_pid= -created_group=false -installation_token= -group_id= expected_uid=0 [[ ${CI_FLEET_TESTING:-0} != 1 ]] || expected_uid=$(id -u) -cleanup() { - local status=$? - trap - EXIT HUP INT TERM - [[ -z $callback_pid ]] || kill "$callback_pid" >/dev/null 2>&1 || true - if $created_group && declare -F rollback_new_group >/dev/null; then - rollback_new_group || printf 'ERROR: automatic rollback of the unverified runner group failed\n' >&2 - fi - rm -rf "$temporary" - exit "$status" -} -trap cleanup EXIT -trap 'exit 129' HUP -trap 'exit 130' INT -trap 'exit 143' TERM if [[ $mode != dry-run && ${CI_FLEET_TESTING:-0} != 1 && ${EUID:-$(id -u)} -ne 0 ]]; then die 'run live/check bootstrap as root'; fi [[ $bind == 127.0.0.1 && $callback_host == 127.0.0.1 ]] || die 'callback bind/host must both be IPv4 loopback 127.0.0.1' @@ -243,7 +270,7 @@ PY note 'Open the URL, authenticate to GitHub, and approve App creation. Return here without copying any value.' deadline=$((SECONDS + timeout)) while [[ ! -f $code_file && $SECONDS -lt $deadline ]]; do - kill -0 "$callback_pid" 2>/dev/null || die 'registration callback stopped before receiving approval' + callback_is_running || { reap_callback_if_exited; die 'registration callback stopped before receiving approval'; } sleep 1 done [[ -f $code_file ]] || die 'registration callback timed out' @@ -295,6 +322,7 @@ verified_app_id=${app_data[1]} rm -f "$app_response" if [[ -n ${handoff_file:-} ]]; then printf 'https://github.com/apps/%s/installations/new\n' "$CI_FLEET_GITHUB_APP_SLUG" >"$handoff_file" + reap_callback_if_exited fi installation_id=${CI_FLEET_GITHUB_APP_INSTALLATION_ID:-} @@ -303,6 +331,7 @@ if [[ -z $installation_id ]]; then note 'Install the App for only the listed private repositories, then return here; polling continues automatically.' deadline=$((SECONDS + timeout)) while ((SECONDS < deadline)); do + reap_callback_if_exited installations=$temporary/installations.json make_app_jwt "$CI_FLEET_GITHUB_APP_ID" "$pem" "$app_jwt" if write_auth_config "$app_jwt" GET https://api.github.com/app/installations "$installations"; then diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh index 1a4d855..dc4e5f1 100755 --- a/scripts/test-bootstrap-github.sh +++ b/scripts/test-bootstrap-github.sh @@ -62,20 +62,20 @@ install_mismatch_root=$tmp/install-mismatch-root if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$install_mismatch_root "$bootstrap" --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --install --config-repo example-org/other --config-ref 1111111111111111111111111111111111111111 >/dev/null 2>&1; then fail 'mismatched installer configuration repository was accepted'; fi [[ ! -e $install_mismatch_root ]] || fail 'mismatched installer repository wrote local state before rejection' config_checkout=$tmp/config-checkout +mkdir "$config_checkout" +cp -a "$repo_root/templates/config-repository/." "$config_checkout/" +fixture_engine_ref=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["controllers"]["example-ci-01"]["engine_ref"])' "$config_checkout/fleet.json") +python3 "$repo_root/templates/config-repository/scripts/init.py" --organization acme-org --project example-repo --repository acme-org/example-repo --runner-group example-ci-experimental --controller example-ci-01 --location example-site --engine-ref "$fixture_engine_ref" --output "$config_checkout/fleet.json" --force git init -q "$config_checkout" -python3 - "$config_checkout/fleet.json" <<'PY' -import json,sys -json.dump({'organization':{'slug':'example-org'},'controllers':{'example-ci-01':{'pool':'trusted'}},'runner_pools':{'trusted':{'runner_group':'example-ci-experimental','allowed_repositories':['example-org/example-repo'],'public_repositories':False}}},open(sys.argv[1],'w')) -PY -git -C "$config_checkout" add fleet.json +git -C "$config_checkout" add -A git -C "$config_checkout" -c user.name=fixture -c user.email=fixture@example.invalid commit -qm fixture config_ref=$(git -C "$config_checkout" rev-parse HEAD) routing_mismatch_root=$tmp/routing-mismatch-root -if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$routing_mismatch_root "$bootstrap" --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group other-group --allow-repository example-org/example-repo=101 --install --config-repo "$config_checkout" --config-ref "$config_ref" >/dev/null 2>&1; then fail 'desired-state routing mismatch was accepted'; fi +if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$routing_mismatch_root "$bootstrap" --organization acme-org --instance example-ci-01 --config-repository acme-org/config --runner-group other-group --allow-repository acme-org/example-repo=101 --install --config-repo "$config_checkout" --config-ref "$config_ref" >/dev/null 2>&1; then fail 'desired-state routing mismatch was accepted'; fi [[ ! -e $routing_mismatch_root ]] || fail 'desired-state routing mismatch wrote local state before rejection' routing_root=$tmp/routing-root routing_port=$(port) -CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$routing_root "$bootstrap" --organization example-org --instance example-ci-01 --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --install --config-repo "$config_checkout" --config-ref "$config_ref" --port "$routing_port" --timeout 30 >"$tmp/routing.out" 2>"$tmp/routing.err" & +CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$routing_root "$bootstrap" --organization acme-org --instance example-ci-01 --config-repository acme-org/config --runner-group example-ci-experimental --allow-repository acme-org/example-repo=101 --install --config-repo "$config_checkout" --config-ref "$config_ref" --port "$routing_port" --timeout 30 >"$tmp/routing.out" 2>"$tmp/routing.err" & routing_pid=$! processes+=("$routing_pid") wait_http "http://127.0.0.1:$routing_port/" "$tmp/routing.html" From 5b7eebb4c4e3d517082113c172237147af2d9fca Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sun, 16 Aug 2026 04:53:07 -0500 Subject: [PATCH 08/10] fix: serialize and bound GitHub bootstrap --- scripts/bootstrap-github-callback.py | 7 +++++ scripts/bootstrap-github.sh | 35 ++++++++++++++++--------- scripts/fixtures/fake-bootstrap-curl.py | 2 ++ scripts/test-bootstrap-github.sh | 18 +++++++++++++ 4 files changed, 50 insertions(+), 12 deletions(-) diff --git a/scripts/bootstrap-github-callback.py b/scripts/bootstrap-github-callback.py index 708d083..4a873f1 100755 --- a/scripts/bootstrap-github-callback.py +++ b/scripts/bootstrap-github-callback.py @@ -1,9 +1,11 @@ #!/usr/bin/env python3 import argparse +import ctypes import html import http.server import os import secrets +import signal import urllib.parse from typing import cast @@ -107,6 +109,11 @@ def main() -> int: args = parser.parse_args() if not 1 <= args.port <= 65535 or not 30 <= args.timeout <= 1800: parser.error("invalid port or timeout") + parent = os.getppid() + if ctypes.CDLL(None, use_errno=True).prctl(1, signal.SIGTERM) != 0: + raise OSError(ctypes.get_errno(), "cannot bind callback lifetime to bootstrap") + if os.getppid() != parent: + return 2 server = Server((args.bind, args.port), Callback) server.organization = args.organization server.state = args.state diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh index 3c9da75..3c25210 100755 --- a/scripts/bootstrap-github.sh +++ b/scripts/bootstrap-github.sh @@ -57,7 +57,7 @@ if $run_installer; then [[ $mode == live ]] || die '--install is available only in live mode' [[ -n $config_repo && $config_ref =~ ^[0-9a-f]{40}$ ]] || die '--install requires --config-repo and a 40-character --config-ref' fi -for command in bash curl openssl python3 install mktemp stat; do command -v "$command" >/dev/null || die "required command is unavailable: $command"; done +for command in bash curl flock openssl python3 install mktemp stat; do command -v "$command" >/dev/null || die "required command is unavailable: $command"; done script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) temporary=$(mktemp -d) callback_pid= @@ -78,11 +78,9 @@ reap_callback_if_exited() { cleanup() { local status=$? trap - EXIT HUP INT TERM - if [[ -n $callback_pid ]]; then - if callback_is_running; then kill "$callback_pid" >/dev/null 2>&1 || true; fi - wait "$callback_pid" >/dev/null 2>&1 || true - callback_pid= - fi + # The callback binds its lifetime to this process with PR_SET_PDEATHSIG; never + # signal a stale numeric PID from cleanup. + reap_callback_if_exited if $created_group && declare -F rollback_new_group >/dev/null; then rollback_new_group || printf 'ERROR: automatic rollback of the unverified runner group failed\n' >&2 fi @@ -137,14 +135,14 @@ if [[ $mode != dry-run && ${CI_FLEET_TESTING:-0} != 1 && ${EUID:-$(id -u)} -ne 0 [[ $bind == 127.0.0.1 && $callback_host == 127.0.0.1 ]] || die 'callback bind/host must both be IPv4 loopback 127.0.0.1' write_auth_config() { - local credential=$1 method=$2 url=$3 output=$4 payload=${5:-} + local credential=$1 method=$2 url=$3 output=$4 payload=${5:-} transfer_timeout=${6:-30} local config=$temporary/curl.$RANDOM.conf - python3 - "$credential" "$method" "$url" "$output" "$payload" "$config" <<'PY' + python3 - "$credential" "$method" "$url" "$output" "$payload" "$config" "$transfer_timeout" <<'PY' from pathlib import Path import sys -credential, method, url, output, payload, target = sys.argv[1:] +credential, method, url, output, payload, target, transfer_timeout = sys.argv[1:] token = Path(credential).read_text().strip() -lines = ['silent', 'show-error', 'fail-with-body', f'request = "{method}"', f'url = "{url}"', +lines = ['silent', 'show-error', 'fail-with-body', 'connect-timeout = "10"', f'max-time = "{transfer_timeout}"', f'request = "{method}"', f'url = "{url}"', f'output = "{output}"', 'header = "Accept: application/vnd.github+json"', 'header = "X-GitHub-Api-Version: 2022-11-28"', f'header = "Authorization: Bearer {token}"'] if payload: @@ -230,6 +228,14 @@ for directory in "$etc_dir" "$secret_dir"; do install -d -m 0700 "$directory" fi done +bootstrap_lock=$etc_dir/bootstrap.lock +if [[ -e $bootstrap_lock || -L $bootstrap_lock ]]; then + [[ -f $bootstrap_lock && ! -L $bootstrap_lock && $(stat -c %a "$bootstrap_lock") == 600 && $(stat -c %u "$bootstrap_lock") == "$expected_uid" ]] || die 'bootstrap lock file is unsafe' +else + install -m 0600 /dev/null "$bootstrap_lock" +fi +exec {bootstrap_lock_fd}<>"$bootstrap_lock" +flock -n "$bootstrap_lock_fd" || die 'another bootstrap transaction is active' if [[ -f $bootstrap_pending && ! -L $bootstrap_pending && $(stat -c %u "$bootstrap_pending") == "$expected_uid" && $(stat -c %a "$bootstrap_pending") == 600 ]]; then if python3 - "$bootstrap_pending" <<'PY' import json,sys @@ -246,6 +252,9 @@ if [[ -e $bootstrap_recovery || -L $bootstrap_recovery ]]; then [[ $mode != check ]] || die 'bootstrap credential recovery is pending; rerun live bootstrap first' recover_conversion fi +if [[ -e $bootstrap_state || -L $bootstrap_state ]]; then + [[ -f $bootstrap_state && ! -L $bootstrap_state && $(stat -c %a "$bootstrap_state") == 600 && $(stat -c %u "$bootstrap_state") == "$expected_uid" ]] || die 'existing bootstrap state is unsafe' +fi if ! load_metadata; then [[ $mode != check ]] || die 'bootstrap state is missing; run live bootstrap first' [[ ! -e $pem && ! -L $pem && ! -e $host_env && ! -L $host_env ]] || die 'existing credentials lack matching bootstrap identity; no replacement made' @@ -281,7 +290,7 @@ import sys code_file, output, config = map(Path, sys.argv[1:]) code = code_file.read_text().strip() if not code or len(code) > 512 or any(ch.isspace() for ch in code): raise SystemExit(1) -config.write_text('\n'.join(['silent','show-error','fail-with-body','request = "POST"', +config.write_text('\n'.join(['silent','show-error','fail-with-body','connect-timeout = "10"','max-time = "30"','request = "POST"', f'url = "https://api.github.com/app-manifests/{code}/conversions"', f'output = "{output}"', 'header = "Accept: application/vnd.github+json"','header = "X-GitHub-Api-Version: 2022-11-28"']) + '\n') PY @@ -334,7 +343,9 @@ if [[ -z $installation_id ]]; then reap_callback_if_exited installations=$temporary/installations.json make_app_jwt "$CI_FLEET_GITHUB_APP_ID" "$pem" "$app_jwt" - if write_auth_config "$app_jwt" GET https://api.github.com/app/installations "$installations"; then + remaining=$((deadline - SECONDS)); ((remaining > 0)) || break + request_timeout=$((remaining < 30 ? remaining : 30)) + if write_auth_config "$app_jwt" GET https://api.github.com/app/installations "$installations" '' "$request_timeout"; then installation_id=$(python3 - "$installations" "$organization" <<'PY' import json,sys for value in json.load(open(sys.argv[1])): diff --git a/scripts/fixtures/fake-bootstrap-curl.py b/scripts/fixtures/fake-bootstrap-curl.py index f142c4d..c9a3e2d 100755 --- a/scripts/fixtures/fake-bootstrap-curl.py +++ b/scripts/fixtures/fake-bootstrap-curl.py @@ -10,6 +10,8 @@ if " = " in line: key, value = line.split(" = ", 1) values.setdefault(key, []).append(value.strip().strip('"')) +if "connect-timeout" not in values or "max-time" not in values: + raise SystemExit("fake API request lacks transfer bounds") url = values["url"][0] endpoint = url.split("?", 1)[0] method = values.get("request", ["GET"])[0] diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh index dc4e5f1..1804d48 100755 --- a/scripts/test-bootstrap-github.sh +++ b/scripts/test-bootstrap-github.sh @@ -81,6 +81,11 @@ processes+=("$routing_pid") wait_http "http://127.0.0.1:$routing_port/" "$tmp/routing.html" kill -TERM "$routing_pid" if wait "$routing_pid"; then fail 'cancelled matching desired-state bootstrap succeeded'; fi +for _ in {1..100}; do + if ! "$real_curl" -fsS "http://127.0.0.1:$routing_port/" -o /dev/null 2>/dev/null; then break; fi + sleep 0.05 +done +if "$real_curl" -fsS "http://127.0.0.1:$routing_port/" -o /dev/null 2>/dev/null; then fail 'callback outlived its bootstrap parent'; fi if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group example-ci-experimental \ --allow-repository example-org/example-repo --bind 10.0.0.1 --callback-host 10.0.0.1 >/dev/null 2>&1; then fail 'plaintext non-loopback callback was accepted' @@ -96,6 +101,13 @@ if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$unsafe_root "$bootstrap" --check --o --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'symlinked protected directory was accepted' fi +unsafe_state_root=$tmp/unsafe-state-root +mkdir -p "$unsafe_state_root/etc/ci-fleet/secrets" +chmod 700 "$unsafe_state_root/etc/ci-fleet" "$unsafe_state_root/etc/ci-fleet/secrets" +mkdir "$unsafe_state_root/etc/ci-fleet/bootstrap-app.env" +if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$unsafe_state_root "$bootstrap" --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'unsafe bootstrap state was accepted'; fi +[[ ! -e $unsafe_state_root/etc/ci-fleet/secrets/github-app.pem ]] || fail 'unsafe bootstrap state was rejected after credential mutation' # Mock the GitHub API while exercising the complete local callback, conversion, persistence, # exact private-repository/group checks, idempotent check, and redaction paths. @@ -137,6 +149,12 @@ if grep -R -F -e fixture-conversion-code -e fixture-client-secret -e fixture-web fail 'bootstrap output exposed sensitive fixture material' fi grep -Fxq 'POST runner-group-create' "$tmp/api.log" || fail 'runner group was not created in the mocked live flow' +exec {held_bootstrap_lock}<>"$live_root/etc/ci-fleet/bootstrap.lock" +flock -n "$held_bootstrap_lock" +if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'concurrent bootstrap transaction was accepted'; fi +flock -u "$held_bootstrap_lock" +exec {held_bootstrap_lock}>&- create_count=$(grep -c '^POST runner-group-create$' "$tmp/api.log") rm -f "$FAKE_BOOTSTRAP_STATE" if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --organization example-org --instance example-ci-01 \ From fb6b12c794c71034349f063d2f128f2ff2bcd4a8 Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sun, 16 Aug 2026 05:14:13 -0500 Subject: [PATCH 09/10] fix: close bootstrap lifecycle races --- scripts/bootstrap-github-callback.py | 10 +++-- scripts/bootstrap-github.sh | 53 +++++++++++++++++-------- scripts/fixtures/fake-bootstrap-curl.py | 8 +++- scripts/test-bootstrap-github.sh | 10 ++++- 4 files changed, 58 insertions(+), 23 deletions(-) diff --git a/scripts/bootstrap-github-callback.py b/scripts/bootstrap-github-callback.py index 4a873f1..45d4eef 100755 --- a/scripts/bootstrap-github-callback.py +++ b/scripts/bootstrap-github-callback.py @@ -27,8 +27,8 @@ class Callback(http.server.BaseHTTPRequestHandler): def app_server(self) -> Server: return cast(Server, self.server) - def log_message(self, _format: str, *_args: object) -> None: - return + def log_message(self, format: str, *args: object) -> None: + del format, args def reply(self, status: int, body: str) -> None: data = body.encode() @@ -99,6 +99,7 @@ def do_GET(self) -> None: def main() -> int: parser = argparse.ArgumentParser() parser.add_argument("--bind", required=True) + parser.add_argument("--parent-pid", required=True, type=int) parser.add_argument("--port", required=True, type=int) parser.add_argument("--organization", required=True) parser.add_argument("--state", required=True) @@ -109,10 +110,11 @@ def main() -> int: args = parser.parse_args() if not 1 <= args.port <= 65535 or not 30 <= args.timeout <= 1800: parser.error("invalid port or timeout") - parent = os.getppid() + if os.getppid() != args.parent_pid: + return 2 if ctypes.CDLL(None, use_errno=True).prctl(1, signal.SIGTERM) != 0: raise OSError(ctypes.get_errno(), "cannot bind callback lifetime to bootstrap") - if os.getppid() != parent: + if os.getppid() != args.parent_pid: return 2 server = Server((args.bind, args.port), Callback) server.organization = args.organization diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh index 3c25210..02d55a4 100755 --- a/scripts/bootstrap-github.sh +++ b/scripts/bootstrap-github.sh @@ -229,11 +229,8 @@ for directory in "$etc_dir" "$secret_dir"; do fi done bootstrap_lock=$etc_dir/bootstrap.lock -if [[ -e $bootstrap_lock || -L $bootstrap_lock ]]; then - [[ -f $bootstrap_lock && ! -L $bootstrap_lock && $(stat -c %a "$bootstrap_lock") == 600 && $(stat -c %u "$bootstrap_lock") == "$expected_uid" ]] || die 'bootstrap lock file is unsafe' -else - install -m 0600 /dev/null "$bootstrap_lock" -fi +if [[ ! -e $bootstrap_lock && ! -L $bootstrap_lock ]]; then (set -o noclobber; : >"$bootstrap_lock") 2>/dev/null || true; fi +[[ -f $bootstrap_lock && ! -L $bootstrap_lock && $(stat -c %a "$bootstrap_lock") == 600 && $(stat -c %u "$bootstrap_lock") == "$expected_uid" ]] || die 'bootstrap lock file is unsafe' exec {bootstrap_lock_fd}<>"$bootstrap_lock" flock -n "$bootstrap_lock_fd" || die 'another bootstrap transaction is active' if [[ -f $bootstrap_pending && ! -L $bootstrap_pending && $(stat -c %u "$bootstrap_pending") == "$expected_uid" && $(stat -c %a "$bootstrap_pending") == 600 ]]; then @@ -273,7 +270,7 @@ PY ) code_file=$temporary/manifest-code handoff_file=$temporary/installation-url - python3 "$script_dir/bootstrap-github-callback.py" --bind "$bind" --port "$port" --organization "$organization" --state "$state" --manifest "$manifest" --output "$code_file" --handoff "$handoff_file" --timeout "$timeout" & + python3 "$script_dir/bootstrap-github-callback.py" --parent-pid "$$" --bind "$bind" --port "$port" --organization "$organization" --state "$state" --manifest "$manifest" --output "$code_file" --handoff "$handoff_file" --timeout "$timeout" & callback_pid=$! note "REGISTRATION_URL http://$callback_host:$port/" note 'Open the URL, authenticate to GitHub, and approve App creation. Return here without copying any value.' @@ -453,18 +450,40 @@ then die 'existing runner group is broader or has unexpected identity; no change made' fi rm -f "$response" -selected=$temporary/group-repositories.json -if ! write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id/repositories?per_page=100" "$selected"; then - if $created_group; then rollback_new_group || die 'new runner group repository inspection and rollback both failed'; die 'runner group repository inspection failed; new group was rolled back'; fi - die 'existing runner group repository inspection failed; no change made' -fi -if ! python3 - "$selected" "$repository_ids" "${repository_names[@]}" <<'PY' +selected_pages=() +selected_total=-1 +selected_seen=0 +page=1 +while :; do + selected=$temporary/group-repositories-$page.json + if ! write_auth_config "$installation_token" GET "https://api.github.com/orgs/$organization/actions/runner-groups/$group_id/repositories?per_page=100&page=$page" "$selected"; then + if $created_group; then rollback_new_group || die 'new runner group repository inspection and rollback both failed'; die 'runner group repository inspection failed; new group was rolled back'; fi + die 'existing runner group repository inspection failed; no change made' + fi + selected_pages+=("$selected") + stats=$(python3 - "$selected" <<'PY' +import json,sys +value=json.load(open(sys.argv[1])); repositories=value.get('repositories'); total=value.get('total_count') +if not isinstance(total,int) or total < 0 or not isinstance(repositories,list) or len(repositories)>100: raise SystemExit(1) +print(total,len(repositories)) +PY + ) || die 'runner group repository page is malformed' + read -r page_total page_count <<<"$stats" + if ((selected_total < 0)); then selected_total=$page_total; else ((page_total == selected_total)) || die 'runner group repository page totals differ'; fi + ((selected_seen += page_count)) + ((selected_seen <= selected_total)) || die 'runner group repository pages exceed total_count' + ((selected_seen < selected_total)) || break + ((page_count > 0)) || die 'runner group repository pagination made no progress' + ((page += 1)) +done +if ! python3 - "$repository_ids" "${repository_names[@]}" -- "${selected_pages[@]}" <<'PY' import json,sys from pathlib import Path -response=json.load(open(sys.argv[1])); repositories=response.get('repositories',[]) -if response.get('total_count') != len(repositories): raise SystemExit(1) -actual=sorted(v['id'] for v in repositories); expected=sorted(int(v) for v in Path(sys.argv[2]).read_text().splitlines()) -actual_names=sorted(v['full_name'].lower() for v in repositories); expected_names=sorted(v.lower() for v in sys.argv[3:]) +separator=sys.argv.index('--'); expected_names=sorted(v.lower() for v in sys.argv[2:separator]); pages=sys.argv[separator+1:] +responses=[json.load(open(path)) for path in pages]; repositories=[repository for response in responses for repository in response['repositories']] +if any(response.get('total_count') != len(repositories) for response in responses): raise SystemExit(1) +actual=sorted(v['id'] for v in repositories); expected=sorted(int(v) for v in Path(sys.argv[1]).read_text().splitlines()) +actual_names=sorted(v['full_name'].lower() for v in repositories) if actual!=expected or actual_names!=expected_names or any(v.get('private') is not True or v.get('archived') is not False for v in repositories): raise SystemExit(1) PY then @@ -474,7 +493,7 @@ then fi die 'existing runner group repository access differs; no change made' fi -rm -f "$selected" +rm -f "${selected_pages[@]}" created_group=false staged=$temporary/host.env diff --git a/scripts/fixtures/fake-bootstrap-curl.py b/scripts/fixtures/fake-bootstrap-curl.py index c9a3e2d..26f55eb 100755 --- a/scripts/fixtures/fake-bootstrap-curl.py +++ b/scripts/fixtures/fake-bootstrap-curl.py @@ -3,6 +3,7 @@ import os from pathlib import Path import sys +import urllib.parse config = Path(sys.argv[sys.argv.index("--config") + 1]).read_text().splitlines() values = {} @@ -51,7 +52,12 @@ response = {"total_count": len(groups), "runner_groups": groups} elif endpoint.endswith("/actions/runner-groups/789/repositories"): kind = "runner-group-repositories" - response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True, "archived": False}]} + page = int(urllib.parse.parse_qs(urllib.parse.urlsplit(url).query).get("page", ["1"])[0]) + if os.environ.get("FAKE_BOOTSTRAP_SECOND_GROUP_PAGE") == "1": + repositories = [{"id": 101, "full_name": "example-org/example-repo", "private": True, "archived": False}] if page == 1 else [{"id": 102, "full_name": "example-org/second-repo", "private": True, "archived": False}] if page == 2 else [] + response = {"total_count": 2, "repositories": repositories} + else: + response = {"total_count": 1, "repositories": [{"id": 101, "full_name": "example-org/example-repo", "private": True, "archived": False}]} elif endpoint.endswith("/actions/runner-groups/789") and method == "DELETE": kind = "runner-group-delete" state.unlink(missing_ok=True) diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh index 1804d48..11343ce 100755 --- a/scripts/test-bootstrap-github.sh +++ b/scripts/test-bootstrap-github.sh @@ -28,7 +28,7 @@ wait_http() { callback_port=$(port) code_file=$tmp/callback-code handoff_file=$tmp/callback-handoff -python3 "$callback" --bind 127.0.0.1 --port "$callback_port" --organization example-org --state fixture-state \ +python3 "$callback" --parent-pid "$$" --bind 127.0.0.1 --port "$callback_port" --organization example-org --state fixture-state \ --manifest '{"name":"fixture"}' --output "$code_file" --handoff "$handoff_file" --timeout 30 >"$tmp/callback.out" 2>"$tmp/callback.err" & processes+=("$!") wait_http "http://127.0.0.1:$callback_port/" "$tmp/callback.html" @@ -41,6 +41,9 @@ wait "${processes[-1]}" [[ $(<"$code_file") == fixture-code && $(stat -c %a "$code_file") == 600 ]] || fail 'callback did not protect the conversion code' [[ ! -s $tmp/callback.out && ! -s $tmp/callback.err ]] || fail 'callback logged request material' grep -Fq 'https://github.com/apps/example-fixture/installations/new' "$tmp/next.html" || fail 'callback did not provide the installation approval link' +orphan_port=$(port) +if python3 "$callback" --parent-pid 1 --bind 127.0.0.1 --port "$orphan_port" --organization example-org --state fixture-state --manifest '{"name":"fixture"}' --output "$tmp/orphan-code" --handoff "$tmp/orphan-handoff" --timeout 30 >/dev/null 2>&1; then fail 'callback accepted a mismatched bootstrap parent'; fi +if "$real_curl" -fsS "http://127.0.0.1:$orphan_port/" -o /dev/null 2>/dev/null; then fail 'orphaned callback opened a listener'; fi # Dry-run validates a complete fictional request and performs no write or API call. dry_root=$tmp/dry-root @@ -79,6 +82,9 @@ CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$routing_root "$bootstrap" --organizatio routing_pid=$! processes+=("$routing_pid") wait_http "http://127.0.0.1:$routing_port/" "$tmp/routing.html" +second_routing_port=$(port) +if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$routing_root "$bootstrap" --organization acme-org --instance example-ci-01 --config-repository acme-org/config --runner-group example-ci-experimental --allow-repository acme-org/example-repo=101 --install --config-repo "$config_checkout" --config-ref "$config_ref" --port "$second_routing_port" --timeout 30 >/dev/null 2>&1; then fail 'concurrent first-time bootstrap was accepted'; fi +if "$real_curl" -fsS "http://127.0.0.1:$second_routing_port/" -o /dev/null 2>/dev/null; then fail 'concurrent bootstrap reached callback registration'; fi kill -TERM "$routing_pid" if wait "$routing_pid"; then fail 'cancelled matching desired-state bootstrap succeeded'; fi for _ in {1..100}; do @@ -173,6 +179,8 @@ PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$boot [[ $(sha256sum "$host_env" "$pem") == "$before" ]] || fail 'check mode changed credentials' grep -Fq 'CREDENTIALS_VERIFIED' "$tmp/check.out" || fail 'check mode did not verify persisted credentials' [[ $(grep -c '^POST runner-group-create$' "$tmp/api.log") == "$post_rollback_create_count" ]] || fail 'check mode mutated the runner group' +PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root FAKE_BOOTSTRAP_SECOND_GROUP_PAGE=1 "$bootstrap" --check --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 --allow-repository example-org/second-repo=102 >/dev/null 2>&1 || fail 'valid second runner-group repository page was rejected' if PATH="$fake_bin:$PATH" CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$live_root "$bootstrap" --check --organization example-org --instance other-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'persisted App was rebound to another instance'; fi From 113bad047d1ea9af4709353d3621329064a4f90f Mon Sep 17 00:00:00 2001 From: Nicks Hermes Date: Sun, 16 Aug 2026 05:31:40 -0500 Subject: [PATCH 10/10] fix: preserve ambiguous recovery state --- scripts/bootstrap-github.sh | 14 +++++++++----- scripts/test-bootstrap-github.sh | 14 ++++++++++++++ 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/scripts/bootstrap-github.sh b/scripts/bootstrap-github.sh index 02d55a4..cbd0de1 100755 --- a/scripts/bootstrap-github.sh +++ b/scripts/bootstrap-github.sh @@ -14,13 +14,13 @@ EOF die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } note() { printf '%s\n' "$*"; } -mode=live; organization=; instance=; runner_group=; config_repository=; bind=127.0.0.1; callback_host=127.0.0.1 +mode=live; mode_selected=false; organization=; instance=; runner_group=; config_repository=; bind=127.0.0.1; callback_host=127.0.0.1 port=8765; timeout=600; run_installer=false; config_repo=; config_ref= repositories=() while (($#)); do case $1 in - --dry-run) mode=dry-run; shift ;; - --check) mode=check; shift ;; + --dry-run) if $mode_selected; then die '--dry-run and --check are mutually exclusive'; fi; mode=dry-run; mode_selected=true; shift ;; + --check) if $mode_selected; then die '--dry-run and --check are mutually exclusive'; fi; mode=check; mode_selected=true; shift ;; --organization) (($# >= 2)) || die '--organization requires a value'; organization=$2; shift 2 ;; --instance) (($# >= 2)) || die '--instance requires a value'; instance=$2; shift 2 ;; --runner-group) (($# >= 2)) || die '--runner-group requires a value'; runner_group=$2; shift 2 ;; @@ -239,7 +239,9 @@ import json,sys value=json.load(open(sys.argv[1])) if any(not value.get(key) for key in ('id','client_id','pem','slug')): raise SystemExit(1) PY - then mv -fT "$bootstrap_pending" "$bootstrap_recovery" + then + [[ ! -e $bootstrap_recovery && ! -L $bootstrap_recovery ]] || die 'pending and recovery records both exist; owner reconciliation is required' + mv -T "$bootstrap_pending" "$bootstrap_recovery" else die "incomplete protected conversion response remains at $bootstrap_pending; do not retry or delete it without owner recovery/abandonment approval" fi elif [[ -e $bootstrap_pending || -L $bootstrap_pending ]]; then @@ -301,7 +303,9 @@ import json,sys value=json.load(open(sys.argv[1])) if any(not value.get(key) for key in ('id','client_id','pem','slug')): raise SystemExit(1) PY - then mv -fT "$bootstrap_pending" "$bootstrap_recovery" + then + [[ ! -e $bootstrap_recovery && ! -L $bootstrap_recovery ]] || die 'pending and recovery records both exist; owner reconciliation is required' + mv -T "$bootstrap_pending" "$bootstrap_recovery" else die "manifest conversion did not produce a complete response (curl status $conversion_status); protected pending bytes retained for owner recovery" fi if [[ ${CI_FLEET_TEST_FAIL_AFTER_CONVERSION:-0} == 1 && ${CI_FLEET_TESTING:-0} == 1 ]]; then die 'injected failure after manifest conversion'; fi diff --git a/scripts/test-bootstrap-github.sh b/scripts/test-bootstrap-github.sh index 11343ce..5d601c1 100755 --- a/scripts/test-bootstrap-github.sh +++ b/scripts/test-bootstrap-github.sh @@ -51,6 +51,10 @@ CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$dry_root "$bootstrap" --dry-run --organ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >"$tmp/dry.out" grep -Fq 'NO_GITHUB_MUTATION' "$tmp/dry.out" || fail 'dry-run omitted its no-mutation result' [[ ! -e $dry_root ]] || fail 'dry-run wrote host state' +conflicting_root=$tmp/conflicting-mode-root +if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$conflicting_root "$bootstrap" --dry-run --check --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'conflicting dry-run/check modes were accepted'; fi +[[ ! -e $conflicting_root ]] || fail 'conflicting mode rejection wrote local state' if "$bootstrap" --dry-run --organization example-org --instance example-ci-01 --runner-group Default --allow-repository example-org/example-repo >/dev/null 2>&1; then fail 'default runner group was accepted' fi @@ -114,6 +118,16 @@ mkdir "$unsafe_state_root/etc/ci-fleet/bootstrap-app.env" if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$unsafe_state_root "$bootstrap" --organization example-org --instance example-ci-01 \ --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'unsafe bootstrap state was accepted'; fi [[ ! -e $unsafe_state_root/etc/ci-fleet/secrets/github-app.pem ]] || fail 'unsafe bootstrap state was rejected after credential mutation' +dual_record_root=$tmp/dual-record-root +mkdir -p "$dual_record_root/etc/ci-fleet/secrets" +chmod 700 "$dual_record_root/etc/ci-fleet" "$dual_record_root/etc/ci-fleet/secrets" +printf '%s\n' '{"id":1,"client_id":"Iv1.pending","pem":"pending","slug":"pending"}' >"$dual_record_root/etc/ci-fleet/bootstrap-recovery.pending" +printf '%s\n' '{"id":2,"client_id":"Iv1.recovery","pem":"recovery","slug":"recovery"}' >"$dual_record_root/etc/ci-fleet/bootstrap-recovery.json" +chmod 600 "$dual_record_root/etc/ci-fleet/bootstrap-recovery.pending" "$dual_record_root/etc/ci-fleet/bootstrap-recovery.json" +dual_before=$(sha256sum "$dual_record_root/etc/ci-fleet/bootstrap-recovery.pending" "$dual_record_root/etc/ci-fleet/bootstrap-recovery.json") +if CI_FLEET_TESTING=1 CI_FLEET_ROOT_PREFIX=$dual_record_root "$bootstrap" --organization example-org --instance example-ci-01 \ + --config-repository example-org/config --runner-group example-ci-experimental --allow-repository example-org/example-repo=101 >/dev/null 2>&1; then fail 'ambiguous dual recovery records were accepted'; fi +[[ $(sha256sum "$dual_record_root/etc/ci-fleet/bootstrap-recovery.pending" "$dual_record_root/etc/ci-fleet/bootstrap-recovery.json") == "$dual_before" ]] || fail 'ambiguous recovery records were overwritten' # Mock the GitHub API while exercising the complete local callback, conversion, persistence, # exact private-repository/group checks, idempotent check, and redaction paths.