Skip to content
Draft
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 96 additions & 93 deletions docs/GITHUB-APP-SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,104 +16,91 @@ organization, app name, and configuration repository.
identity, not a personal account.
- Revoking access is removing an installation, not rotating a user's keys.

## 1. Create the app
## Bootstrap on the target host

Organization Settings → Developer settings → GitHub Apps → New GitHub App.

| Field | Value |
| --- | --- |
| Name | one per controller, e.g. `ci-fleet-<controller-id>` |
| Homepage URL | your organization or fleet repository URL |
| Webhook | disabled — the controller polls; nothing calls it |

## 2. Grant minimum permissions

Repository permissions:

| Permission | Access | Why |
| --- | --- | --- |
| Contents | Read-only | fetch desired-state configuration over HTTPS |

Organization permissions:

| Permission | Access | Why |
| --- | --- | --- |
| Self-hosted runners | Read & write | mint runner registration tokens |

Nothing else. No `write` on contents, no actions, no administration. If the
controller ever needs more, that is a reviewed design change, not a settings
tweak.

## 3. Generate and transfer the private key

On the app page: Private keys → Generate a private key. GitHub downloads one
PEM. The normal manual workflow in this guide stores it on the controller's
local filesystem at `/etc/ci-fleet/secrets/github-app.pem`, owned by root with
mode `0600`.

GitHub's browser download is necessarily present briefly on a controlled
management workstation. Choose a fresh temporary directory outside synchronized,
indexed, and backed-up locations, restrict the downloaded file to the operator
immediately, and transfer it at once. This is transient handling, not approved
long-term credential storage; never claim that the key was absent from the
workstation.

Before any key bytes arrive, create the controller directory as root-owned mode
`0700` and pre-create the destination as a root-owned regular file with mode
`0600`. Initial setup uses only the active path:
Run the reviewed checkout on the target Linux Docker host. The plaintext callback
is loopback-only; non-loopback callbacks are rejected because the one-time
manifest conversion code requires transport confidentiality.

```bash
sudo install -d -o root -g root -m 0700 /etc/ci-fleet/secrets
sudo install -o root -g root -m 0600 /dev/null \
/etc/ci-fleet/secrets/github-app.pem
sudo ./scripts/bootstrap-github.sh \
--organization example-org \
--instance example-ci-01 \
--config-repository example-org/config \
--runner-group example-ci-experimental \
--allow-repository example-org/example-repo=123456
```

Verify the directory and destination ownership, type, and mode without reading
the content. Then use an authenticated encrypted channel to stream into that
already secured file; the transfer must not replace it with a default-mode node.
Keep key bytes out of tracing, logs, stdout, process arguments, Git, issues, and
PRs. Compare a SHA-256 digest at both ends without printing file content, then
verify the destination again.

Delete the workstation copy immediately after authenticated transfer and those
transfer checks succeed, before token, reconciliation, health, or convergence
checks. Stop if transfer verification or local deletion fails. The PEM must
never be committed or printed; see [SECRETS.md](SECRETS.md).

This manual workflow permits exactly two host-local files: the active path above
and `/etc/ci-fleet/secrets/github-app.next.pem` while rotating. It does not cover
other custom paths, symlinks, or an external secret manager's import, rotation,
or deletion lifecycle. Those cases require provider-specific tested automation.
Do not improvise them from these Markdown examples;
[issue #27](https://github.com/RandomDevelopment/ci-fleet/issues/27) tracks that
automation.

## 4. Install the app

App page → Install App → choose the organization → **Only select
repositories**: pick only the private desired-state configuration repository.
Installing on all repositories defeats the permission scoping.

Record from the installation page URL and app page:

- **Client ID** (app page, `Iv1...` / `Iv23...`)
- **Installation ID** (the number at the end of the installation URL)

## 5. Wire the host

`/etc/ci-fleet/host.env` (root-owned `0600`, never committed):
All names above are fictional. The script prints one non-secret local
`REGISTRATION_URL`. Open it, press the single registration button, install the
new App for **only** the private configuration repository, and return to the
terminal. The target host receives and exchanges the temporary code itself.
Comment thread
Nickfost marked this conversation as resolved.
Neither the code nor any credential is copied through a phone, clipboard, chat,
email, issue, or second computer.

For a headless remote host, establish an authenticated SSH local forward from
the management workstation first: `ssh -L 8765:127.0.0.1:8765 HOST`. Keep that
session open, run bootstrap on `HOST`, and open `http://127.0.0.1:8765/` on the
workstation. Close the SSH session immediately after bootstrap. Do not expose
the callback listener or forward on a shared workstation.

Each project argument includes its numeric repository ID, obtained and reviewed
by the organization owner. The App itself receives access only to the separate
configuration repository; the project name/ID pairs scope runner-group routing
without granting the controller App project source access.

The bootstrap:

- requests only `contents: read`, metadata read, and organization self-hosted
runner write permission;
- creates a private, independently revocable App identity for the host;
- verifies callback state and expires the callback after 30–1800 seconds;
- writes the PEM directly to `/etc/ci-fleet/secrets/github-app.pem` and the
client/installation IDs to `/etc/ci-fleet/host.env`, root-owned mode `0600`;
- rejects public, archived, wrong-organization, broader App installation, and
default/broad runner-group access;
- creates a missing selected-repository runner group, but never changes an
existing group whose identity or access differs. A group created by the
current invocation is automatically deleted if inspection, routing
verification, or cancellation occurs before verification completes;
- destroys the conversion code, JWTs, installation token, callback state, and
temporary curl configurations on exit. After a successful conversion, a
protected mode-`0600` `bootstrap-recovery.json` (or atomic-publication
`bootstrap-recovery.pending`) is deliberately retained until the PEM and
identity record are installed. Rerun the identical live command to recover.
If bootstrap is abandoned, an organization owner must first revoke/delete the
exact newly created App, then explicitly remove those recovery files; ordinary
cleanup never discards potentially unique credentials.

Inspect a request without local writes or GitHub calls:

```bash
CI_FLEET_GITHUB_APP_CLIENT_ID=<client id>
CI_FLEET_GITHUB_APP_INSTALLATION_ID=<installation id>
CI_FLEET_GITHUB_APP_PRIVATE_KEY_FILE=/etc/ci-fleet/secrets/github-app.pem
sudo ./scripts/bootstrap-github.sh --dry-run \
--organization example-org --instance example-ci-01 \
--config-repository example-org/config \
--runner-group example-ci-experimental \
--allow-repository example-org/example-repo=123456
```

After success, rerun the same command with `--check` to verify App ownership,
permissions, exact selected private repositories, exact runner-group access,
and host-local credential modes without changing the App or runner group.

Pass `--install --config-repo /PATH/TO/PINNED/CHECKOUT --config-ref REVIEWED_COMMIT`
on the initial live command for direct handoff to the idempotent host installer.
Bootstrap requires the commit in that local checkout and verifies that its
selected controller pool exactly matches the requested organization, runner
group, and project allowlist before any GitHub mutation. Remote `OWNER/REPO`
handoff is rejected because a fresh host has no independently verified Git
credential before App creation. Without `--install`, the redacted final report
prints the installer shape. Externally provisioned credentials remain supported;
the generic installer never requires this bootstrap on every run.

The controller exchanges a short-lived JWT signed with the PEM for an
installation token at runtime (`scripts/github-app-token.sh`). No token is
stored.

## 6. Verify
## Verify

The token helper writes a token to stdout. Every verification invocation must
redirect stdout to `/dev/null`; exit status alone is the result.
Expand Down Expand Up @@ -152,21 +139,37 @@ sudo /opt/ci-fleet/manager/current/scripts/remote-reconcile.sh --check-only
- 401 on token exchange: `host.env` has the wrong client ID, installation ID,
or PEM path.

## Accidental creation rollback

The bootstrap never replaces or broadens an existing App or runner group. A new
runner group is automatically deleted only when the same invocation cannot
verify its identity/access or is cancelled before verification; it never deletes
a pre-existing group. If another new object was approved accidentally, stop
before running the installer. An
organization owner must compare the App slug/ID and group name in the redacted
bootstrap report with GitHub's settings, verify the new group has no runners,
then remove only those exact newly created objects. Preserve and investigate any
pre-existing or mismatched object. App deletion and deletion of any group outside
the bounded automatic rollback above are live GitHub-setting mutations and
therefore require separate authorization; this command does not automate them.

## Rotation

Use this ordered safety checklist for the normal host-local root-owned PEM
workflow. It is a set of gates, not a copy-and-paste shell program.

1. Generate a new GitHub key and transfer, verify, and install it as described
above at the one approved replacement path,
1. Before any key bytes arrive, use a controlled management workstation and
pre-create the one approved replacement path,
`/etc/ci-fleet/secrets/github-app.next.pem`. Pre-create it as root-owned
`0600` inside the root-owned `0700` directory before transfer. Keep the old
key and `/etc/ci-fleet/secrets/github-app.pem` active for rollback, and delete
the workstation copy before continuing.
`0600` inside the root-owned `0700` directory. Generate a new GitHub key,
transfer and verify it there, and keep the old key and
`/etc/ci-fleet/secrets/github-app.pem` active for rollback. Delete the
workstation copy immediately after verified transfer and before token,
reconciliation, health, or convergence checks.
2. Update the protected controller identity configuration to select the new PEM.
3. Activate the new key and require the installed manager's token verification,
reconciliation, health check, and installed-state convergence check all to
succeed with the new key.
3. Require new-key activation, reconciliation, health, or convergence checks
all to succeed with the new key, including the installed manager's token
verification and installed-state convergence check.
4. Confirm the controller remains healthy and converged after a fresh check.
5. Only then revoke the old key in GitHub. Remove its exact old controller PEM
only after revocation is confirmed. Retain the now-active replacement PEM;
Expand Down
126 changes: 126 additions & 0 deletions scripts/bootstrap-github-callback.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
#!/usr/bin/env python3
import argparse
import html
import http.server
import os
import secrets
import urllib.parse
from typing import cast


class Server(http.server.HTTPServer):
organization: str
state: str
manifest: str
output: str
handoff: str
done: bool


class Callback(http.server.BaseHTTPRequestHandler):
server_version = "ci-fleet-bootstrap"
sys_version = ""

@property
def app_server(self) -> Server:
return cast(Server, self.server)

def log_message(self, _format: str, *_args: object) -> None:
return

def reply(self, status: int, body: str) -> None:
data = body.encode()
self.send_response(status)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(data)))
self.send_header("Cache-Control", "no-store")
self.send_header("X-Content-Type-Options", "nosniff")
self.send_header("X-Frame-Options", "DENY")
self.send_header("Content-Security-Policy", "default-src 'none'; form-action https://github.com; frame-ancestors 'none'")
self.end_headers()
self.wfile.write(data)

def do_GET(self) -> None:
parsed = urllib.parse.urlsplit(self.path)
if parsed.path == "/" and not parsed.query:
action = f"https://github.com/organizations/{urllib.parse.quote(self.app_server.organization, safe='')}/settings/apps/new?state={urllib.parse.quote(self.app_server.state, safe='')}"
body = (
"<!doctype html><meta name=viewport content='width=device-width'>"
"<title>Register ci-fleet GitHub App</title>"
"<h1>Register this host's ci-fleet App</h1>"
f"<form action='{html.escape(action, quote=True)}' method=post>"
f"<input type=hidden name=manifest value='{html.escape(self.app_server.manifest, quote=True)}'>"
"<button type=submit>Continue to GitHub</button></form>"
)
self.reply(200, body)
return
if parsed.path == "/next" and not parsed.query:
try:
target = open(self.app_server.handoff, encoding="utf-8").read().strip()
except FileNotFoundError:
self.reply(200, "<meta http-equiv=refresh content='1;url=/next'><p>Preparing the installation approval link…</p>")
return
if not target.startswith("https://github.com/apps/") or not target.endswith("/installations/new"):
self.reply(500, "Invalid installation target")
return
self.reply(200, f"<h1>App created</h1><p><a href='{html.escape(target, quote=True)}'>Continue to GitHub installation approval</a></p>")
self.app_server.done = True
return
if parsed.path != "/callback":
self.reply(404, "Not found")
return
try:
values = urllib.parse.parse_qs(parsed.query, strict_parsing=True)
except ValueError:
self.reply(400, "Invalid callback")
return
if set(values) != {"code", "state"} or any(len(value) != 1 for value in values.values()):
self.reply(400, "Invalid callback")
return
if not secrets.compare_digest(values["state"][0], self.app_server.state):
self.reply(403, "Invalid callback state")
return
code = values["code"][0]
if not code or len(code) > 512 or any(character.isspace() for character in code):
self.reply(400, "Invalid callback")
return
try:
descriptor = os.open(self.app_server.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
except FileExistsError:
self.reply(409, "Callback already consumed")
return
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
handle.write(code)
self.reply(200, "<meta http-equiv=refresh content='1;url=/next'><h1>Registration received</h1><p>Preparing installation approval. No value needs to be copied.</p>")


def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--bind", required=True)
parser.add_argument("--port", required=True, type=int)
parser.add_argument("--organization", required=True)
parser.add_argument("--state", required=True)
parser.add_argument("--manifest", required=True)
parser.add_argument("--output", required=True)
parser.add_argument("--handoff", required=True)
parser.add_argument("--timeout", type=int, default=600)
args = parser.parse_args()
if not 1 <= args.port <= 65535 or not 30 <= args.timeout <= 1800:
parser.error("invalid port or timeout")
server = Server((args.bind, args.port), Callback)
server.organization = args.organization
server.state = args.state
server.manifest = args.manifest
server.output = args.output
server.handoff = args.handoff
server.done = False
server.timeout = 1
deadline = __import__("time").monotonic() + args.timeout
while __import__("time").monotonic() < deadline and not server.done:
server.handle_request()
server.server_close()
return 0 if server.done else 2


if __name__ == "__main__":
raise SystemExit(main())
Loading